Replacing variable length file paths in a GEDCOM file

Neil Schipper

Just to be clear:
Find: (?-s)(?<=^\d FILE /).+(?=/)
Replace: Smith_1234
Mode=regex; checkbox to the right doesn’t matter anymore

Then, Find Next to observe expected matching, Replace to perform the substitution on the currently selected match, Replace All to perform the substitution on entire file.

I have tested this. Are you sure you aren’t acting on a file that has already had the substitutions done?

Neil Schipper

@alan-kilborn said:

you changed (my) original usage of .+ to .*

That’s not how it actually went. I devised my solution independently from reading the OP (deciding early that it should have a look-behind and a look-ahead). Then I saw you had a solution that was satisfactory, and I saw no reason to pipe in.

Much later I noticed in the follow-up convo that @web-master was trying to reduce complexity for his non-technical volunteers, and realized my solution was favorable to that. And it was after that that I realized our solutions also differed in the modifier, and that the asterisk was maybe a bit too loose (although, dollars to doughnuts, they perform equally).

Alan Kilborn

@web-master said in Replacing variable length file paths in a GEDCOM file:

Is there a way to put this into a macro so that at runtime the user is prompted for the replacement string?

As @PeterJones mentioned, not possible with a macro.

But as a PythonScript it is rather simple:

replacement_text = notepad.prompt('Replacement text:', '', '')
if replacement_text:
    find_regex = r'(?-s)(?<=^\d FILE /).+(?=/)'
    editor.rereplace(find_regex, replacement_text)

You’d run the script and then get prompted with:

After pressing OK the replacements would be made.

Web Master

@neil-schipper Yes, I copied and pasted the expression direct from here and the file had def not been processed already.

I don’t know what the protocol is here. May I PM you the file so you can try it?

Alan Kilborn

@web-master

Why don’t you just post a line – or a few lines – here that you think it should match, but it isn’t? That may be more instructive (for those reading along) than taking it offline into some private discussion.

While initially impressed with your initiative, I’m now losing faith in your debugging skills. :-(

Neil Schipper

@web-master Does PM mean the chat? Sure go ahead, but I’m not sure if it supports file transfer. Another option is to put file on a hosting site so we can all access it.

An easy thing to do is to gather up at least a few screens worth of “interesting” data (stressful/challenging to the algorithm) and tossing it into a “literal text” box as you’ve already learned to do (I’m not sure of upper size limit). (Here’s a trick: to defeat inappropriate colorization, add txt immediately after the opening line’s triple-backquotes).

But over and above all of this, do notice that Alan’s post above provides a solution that is everything you dreamed of. You’ll need to install the Pythonscript plugin and learn a bit about making / saving / running a script (I’ve seen step-by-step recipes; I expect someone will link you to a reliable up-to-date one).

Have fun.

Web Master

@alan-kilborn Hi Alan

I have to be circumspect about what I put in a public domain because a GEDCOM file contaions personal; info of living people so there’s a GDPR issue. But, sure I can cut out some of the lines that are giving trouble and post them here. One moment…

Web Master

@web-master OK here are a few lines that are not following Neil’s rule. Note NONE of the lines in the 4,000 line file were changed. It said there were zero occurrences. I copied and pasted the query so it shouldn’t be operator error, but PICNIC problems can’t be ruled out! :)

2 FILE ~/Pictures/Reunion Pictures/Imported Media/HOBDAY, Audrey Lilian.jpg
2 FILE ~/Pictures/Reunion Pictures/Imported Media/HOBDAY, Audrey Lilian - 1.jpg
2 FILE ~/Pictures/Reunion Pictures/Imported Media/HOBDAY, Audrey Lilian - 2.jpg
2 FILE ~/Pictures/Reunion Pictures/Imported Media/Robert Frederic & Audrey Lilian - 3.jpg
2 FILE ~/Pictures/CUMBERLAND BMD/Audrey Cumberland Death Certificate.jpeg
2 FILE ~/Pictures/HOBDAY/Mums School Report.jpg
2 FILE ~/Pictures/CUMBERLAND Parish Registers/Frederick Charles Cumberland Baptism 1901.jpg
2 FILE ~/Pictures/CUMBERLAND BMD/Frederick Cumberland Death 1985.jpeg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0005.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0332.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0370.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0003.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0150.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0014.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0080.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0141.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0172.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0254 (2).jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Dance Programme 1.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Dance Programme 2.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Dance Programme 3.jpg
2 FILE ~/Pictures/CUMBERLAND BMD/Frederick Cumberland Cremation 1985.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Dance Programme 4.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/DSCN0101.jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/DSCN0102.jpg
2 FILE ~/Pictures/CUMBERLAND CENSUS/Frederick Cumberland 1921 Census.jpg
2 FILE ~/Pictures/CUMBERLAND CENSUS/Frederick Cumberland 1921 Census (1).jpg
2 FILE ~/Pictures/CUMBERLAND DAVIS PHOTOS/Cumberland L0108.jpg

Alan Kilborn

@web-master

My 30ms reaction to this data: The ~ is a new, never mentioned before feature of your data. Of course the expressions provided before don’t match it. But you already know this.

Web Master

@alan-kilborn I mentioned in my OP - bullet 2

Alan Kilborn

@web-master said in Replacing variable length file paths in a GEDCOM file:

I mentioned in my OP - bullet 2

It’s true; you did:

The old file path always starts at char 8 in the string but may be: \ / ~ or a drive letter, so position 8 is the most reliable start point

But everyone missed it. :-)
Because we are more focused on sample data and what it looks like.
I guarantee it wouldn’t have been missed if it was provided as part of the original sample data.

Web Master

@alan-kilborn I shall breakout the sackcloth and ashes. :)

Alan Kilborn

@web-master

Possibly the expression you now want (OK, you wanted it from the start!) is:

(?-s)(?<=^\d FILE )~?/.+(?=/)

BTW, here’s the English explanation of it, for maximum learning:

Use these options for the whole regular expression (?-s)
- (hyphen inverts the meaning of the letters that follow) -
- Dot doesn’t match line breaks s
Assert that the regex below can be matched ending at this position (positive lookbehind) (?<=^\d FILE )
Match the character “~” literally ~?
- Between zero and one times, as many times as possible, giving back as needed (greedy) ?
Match the character “/” literally /
Match any single character that is NOT a line break character (line feed, carriage return, form feed, next line, line separator, paragraph separator) .+
- Between one and unlimited times, as many times as possible, giving back as needed (greedy) +
Assert that the regex below can be matched starting at this position (positive lookahead) (?=/)
- Match the character “/” literally /

Created with RegexBuddy

Neil Schipper

@alan-kilborn said in Replacing variable length file paths in a GEDCOM file:

~?

New spec says we need to handle both kinds of slash, tilde, and drive letter, so it needs to be more complex than that, right?

Annoyingly, since the behind text is now varlen, this demolishes the glory of replace text being pure user-text.

Instead of mucking about with all the variations of drive letters and slashes and tildes, I’m gonna propose a looser spec:

following <space after “FILE”>, match any run of text, min len 1, until trailing forw slash

Hence:

Find (?-s)(?<=^\d FILE ).+(?=/)
Replace: /Smith_1234
So replacement text now needs the starting slash.
To keep the user experience elegant, this would be incorp’d into the script:

editor.rereplace(find_regex, '/' + replacement_text)

Web Master

@alan-kilborn Thanks Alan. I think I need to read a regex primer before I have another read of that explanation.

But to go back to my OP and second bullet, the ~ was not the only char to put a spanner in the works. It’s possibe it could also be:

\
Drive letter C: D: etc case insensitive)

… and these appear to trip up Neil’s enhanced version (EDIT: Ah, this message crosses with Neil’s latest, where he identifies the same issue)

But your orginal query handles them all.

I think it’s time to be realistic and say that we have solved it and that the effort to make it completely foolproof is perhaps a stretch farther than we NEED to go

Meanwhile I’ll go off and explore Python

Thank you everyone. It’s only 24 hours since my initial request for help and you’ve all been fab!

Alan Kilborn

@neil-schipper said in Replacing variable length file paths in a GEDCOM file:

New spec says we need to handle both kinds of slash, tilde, and drive letter, so it needs to be more complex than that, right?

Yea.
It’s actually old-spec, though. Maybe let’s call it we-didn’t-read-spec.
:-)

And TBH, my “exactness” tends to fall off the longer these types of threads go on…

However, I think we’ve given the OP enough to get his task carried out, so that’s a good thing.

Web Master

@neil-schipper said in Replacing variable length file paths in a GEDCOM file:

Find (?-s)(?<=^\d FILE ).+(?=/)
Replace: /Smith_1234
So replacement text now needs the starting slash.

That’s the one! And to echo Alan’s last comment, you’ve given me all I asked for and more than enough to be going on with. I’m good to go now, thank you.

Neil Schipper

@alan-kilborn said in Replacing variable length file paths in a GEDCOM file:

It’s actually old-spec, though.

True.

It’s actually old-spec, though. Maybe let’s call it we-didn’t-read-spec.

I’m not so sure about that: Your original expression did pick up the total requirement. My “new and improved” one introduced the non-compliance (and my most recent is in essence the same as your original).

Alan Kilborn

@neil-schipper said in Replacing variable length file paths in a GEDCOM file:

Your original expression did pick up the total requirement.

I think that might have been blind luck. :-)

Paul Wormer

@web-master said in Replacing variable length file paths in a GEDCOM file:

It took long enough on a file with 4K lines but some of the files have 1million lines!!!

I am sorry to bud in, but I noticed that your question is about manually typing in replacement strings. Are you certain you want to do this for 1 million lines? Or is the number of lines starting with 1 FILE small enough to do the typing in a finite time span? Just curious.