Finding and replacing all data between two points?

Joey Flaig

@Ekopalypse Thank you for the insight!

@guy038 And omg this did all 5,145 instances with one click… You just saved me HOURS (days if it weren’t for find and replace) of work! Thank you so completely much!!! Now I can process the rest of the transcript easy as pie too!!! <3

Joey Flaig

@guy038
Since you’re already here, quick unrelated question - since this was formatted for a certain aesthetic, sentences cut off and continue on a new line past a certain point.
Like:
<CLT 4>The entrance is still blocked by that giant hunk of
metal.<CLT>
Would there be any way to find and replace those line breaks? I’m using some direct quotes from the transcript in my writings, and it’d save a lot of frustration to auto-format them instead of doing it manually for every single dialogue that goes on for more than a line. So it’d look like this:
<CLT 4> The entrance is still blocked by that giant hunk of metal. <CLT>

guy038

Hi, @joey-flaig, @ekopalypse and All,

Glad to see that this S/R saved you some hours of work ;-))

No problem, to solve this new and easy challenge, Joey !

So, this could be translated, in current language, as:

Replace any end of line character(s) with a space character only if the following line :

Does not start with the string <CLT 4>
Does end with the string <CLT>

This leads to the regex solution :

SEARCH (?-is)\R(?=(?!<CLT 4>).+<CLT>$)

REPLACE \x20

Notes :

First, the in-line modifiers (?-si) :
- Forces to regex engine to interpret any dot ( . ) as representing a single standard character only, NOT EOL character(s)
- Forces the search to be processed in a non-insensitive way
Then, the \R syntax will match any EOL ( \r\n in Windows files, \n in Unix files and \r in Mac files )
But only if the look-ahead - condition (?=.+<CLT>$) is true, i.e. if the following line ends with the string <CLT>
And only if the second negative nested look-ahead (?!<CLT 4>), right after \R, is also true, i.e. the following line does not begin with the string <CLT 4>

Best Regards,

guy38

Joey Flaig

@guy038 said:
Oh, this helps! I forgot to clarify - stuff marked by <CLT> is just colored text to denote in game importance or the protagonist’s internal thoughts and most of the dialogue doesn’t actually have it, it just looks like this:

<SPEAKER N°004>MONOKUMA</SPEAKER N°004>
<ORIGINAL N°004>
Well then. Since you’re all giving it your best, your
generous headmaster will give you a little hint!

So your solution got some, but not all!

guy038

Hi, @joey-flaig,

You said :

So your solution got some, but not all!

I don’t understand exactly what you mean ! In case, the Multi-lignes, or not, part <CLT 4>...........<CLT> is totally absent, as below, my previous regex ( which I slightly changed, BTW ! ) does not match any EOL char, anyway !?

<SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
<ORIGINAL N°001>
</ORIGINAL N°001>
<JAPANESE N°001>
<CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
</JAPANESE N°001>
<TRANSLATED N°001>

</TRANSLATED N°001>
<COMMENT N°001>
</COMMENT N°001>

BR

guy038

Joey Flaig

Ah, I’m sorry for the confusion! What I mean is that not all dialogue in the game is between <CTL> coding. The regex you made to gets rid of the line break in the dialogue, but only when the dialogue was preceded by <CTL>. So said regex doesn’t remove the line break for all the dialogue, just the portions that are preceeded by <CTL>. I’m looking to remove all dialogue line breaks. Does that make more sense? I can show you screencaps if needed!

guy038

Hello, @@joey-flaig and All,

Ah, OK ! If you want to wipe out any empty line or any empty line, containing blank characters, two solutions :

Use the built-in N++ commands Edit > Line Operations > Remove Empty Lines or Edit > Line Operations > Remove Empty Lines (Containing Blank characters)
Use the search regexes, accordingly, (^\R)+ or ^(\h*\R)+ and leave the Replacement zone EMPTY

BR
guy038

Joey Flaig

@guy038
Oooh, both these solutions are really handy! They don’t quite work for what I’m looking for though, because there aren’t empty lines, but rather unnecessary line breaks in the middle of sentences. So they didn’t do anything.

Like this

After playing around with the Line Operations feature, I found one called Join Lines. If I select both lines 32 and 33 and hit Join Lines, it gives me the desired effect of merging them into one.

Now that I know what NPP+ can do a little bit better, I’ve come up with a more precise way of speaking. Here’s what I want: I’m looking for some way to automatically Join Lines in places where dialogue is more than one line (and therefore contains a line break/Return). The text that would be checked for this always starts on the line after the <ORIGINAL N°00#> line. Otherwise it would join the coding, SPEAKER, and ORIGINAL line with all the dialogue automatically and that would be a pain to take apart later). Some kind of process or find and replace would be ideal for this because there are thousands of lines I’d have to manually look through.

This seems to be getting more and more complicated! Does this make more sense now?

guy038

Hi @@joey-flaig and All,

Ah, I clearly see, now, what you’re looking for ;-)) Actually, you don’t mind about the tags <CLT 4> and <CLT> at all !

So, the magic regex is :

SEARCH (?-si)^<ORIGINAL N°\d+>\R.+?\K\h*\R\h*(?!<)

REPLACE \x20

Tick, preferably, the Wrap around option
Select the Regular expression search mode
Due to the \K syntax, you must use the Replace All button, exclusively

Et voilà !

Notes :

First, as usual, the in-line modifiers (?s-i) :
- Forces to regex engine to interpret any dot ( . ) as representing any single character, even EOL characters, (?-s)
- Forces the search to be processed in a non-insensitive way, (?-i)
Then, from beginning of line ( ^ ), the regex searches for the string <ORIGINAL N°, followed with some digits ( \d+ ), followed with the > symbol, closed to its EOL chars ( \R)
Now, on the line, following the line <ORIGINAL N°###>, it looks for the shortest range of standard characters ( .+? )
At this point, the \K syntax resets the regex engine search and position. So, from now on, the final search looks for possible blank characters followed with EOL chars, followed, again, with possible blank chars ( \h*\R\h* )
But this search happens ONLY IF the negative look-ahead (?!<) is verified, i.e. if the next line does not begin with the < symbol ( => is, simply, the continuation text of the previous line )
In that case, the EOL characters, possibly surrounded with blank characters, are replaced with a single space character ( \x20 )

Best Regards,

guy038

P.S. :

The fact of searching for \h*\R\h* and replacing the \x20 ensures you that the two parts of the text will be always joined with an unique space character ;-))
If, between the tags <ORIGINAL N°###> and </ORIGINAL N°###>, you have more than 2 lines of text, like below :

<ORIGINAL N°001>
<CLT 4>The entrance
    is still blocked
        by that
giant
   hunk of
metal.<CLT>
</ORIGINAL N°001>

No problem ! Just click, several times, on the Replace All button, until the message Replace All: 0 occurrences were replaced. occurs ;-))

And you’ll be left with :

<ORIGINAL N°001>
<CLT 4>The entrance is still blocked by that giant hunk of metal.<CLT>
</ORIGINAL N°001>

Joey Flaig

Yeah, this is what I was looking to do! This regex looks great… but it sometimes fuses the line indicating a speaker change [the ones with all the dashes that look like ----------------) with the dialogue! Any way to tweak the regex to prevent that?

https://imgur.com/OhYLhBM

If it matters, the speaker change lines contain 60 dashes!

guy038

Hi, @@joey-flaig and All,

I’ve got no much time, right now ! So try the exact regex, below :

SEARCH (?-si)^<ORIGINAL N°\d+>\R.+?\K\h*\R\h*+(?!<|-{5,})

REPLACE \x20

I’ll explain to you when I’m back, about six hours later !

Cheers,

guy038

Alan Kilborn

@guy038 said:

I’ve got no much time, right now

And it seems like you could have a full time job, writing this person’s regexes!

guy038

Hello, @@joey-flaig and All,

So yesterday, my wife and I were accompanying 7/8 years old children ( our daughter’s class +2 other classes ), on a school trip and then, in the late afternoon, I preferred to watch the USA-Spain match of the women’s world football championship ! Of course, I wanted to know who France’s opponent would be ;-))

Ah ! I understood why the separation line, made up of 60 dashes, was missing in the original Joey’s post ! Just because, due to the MarkDown syntax, any line of, at least, 3 dashes produces a slight gray horizontal rule, instead ;-))

So, Joey, just forget, my quick previous post and let’s recapitulate all the process :

We’ll start with this sample text of 4 blocks, between the lines of dashes. In two of them, I placed some multi-lines text, inside the section <CLT 4>.............<CLT>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
<ORIGINAL N°001>
<CLT 4>Just some   
    
   
     text to test
my new
regex !   
<CLT>
</ORIGINAL N°001>
<JAPANESE N°001>
<CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
</JAPANESE N°001>
<TRANSLATED N°001>

</TRANSLATED N°001>
<COMMENT N°001>
</COMMENT N°001>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
<ORIGINAL N°002>
<CLT 4>This mailbox doesn’t seem important right now.<CLT>
</ORIGINAL N°002>
<JAPANESE N°002>
<CLT 4>このレターケースは…
今回の事件には関係ないよな…<CLT>
</JAPANESE N°002>
<TRANSLATED N°002>

</TRANSLATED N°002>
<COMMENT N°002>
</COMMENT N°002>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
<ORIGINAL N°003>
<CLT 4>The entrance is still blocked by that giant hunk of
metal.<CLT>
</ORIGINAL N°003>
<JAPANESE N°003>
<CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
</JAPANESE N°003>
<TRANSLATED N°003>

</TRANSLATED N°003>
<COMMENT N°003>
</COMMENT N°003>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
<ORIGINAL N°004>
<CLT 4>An other   
     text to see
how the
regex works<CLT>
</ORIGINAL N°004>
<JAPANESE N°004>
<CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
</JAPANESE N°004>
<TRANSLATED N°004>

</TRANSLATED N°004>
<COMMENT N°004>
</COMMENT N°004>

Using the regex S/R, below, we keep the interesting part of each block, from Joey’s point of view, of course !

SEARCH (?s-i)^\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\1>\R

REPLACE Leave EMPTY

And we get :

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
<ORIGINAL N°001>
<CLT 4>Just some   
    
   
     text to test
my new
regex !   
<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
<ORIGINAL N°002>
<CLT 4>This mailbox doesn’t seem important right now.<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
<ORIGINAL N°003>
<CLT 4>The entrance is still blocked by that giant hunk of
metal.<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
<ORIGINAL N°004>
<CLT 4>An other   
     text to see
how the
regex works<CLT>

Now, in order to change all multi-lines sections <CLT 4>.............<CLT> into a mono-line one, use the following regex S/R :

SEARCH (?-s)(?:(?!<|>|-).)+?\K(?:\h*\R\h*)++(?=(<CLT>)|.+)

REPLACE ?1:\x20

And here is your expected text :

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
<ORIGINAL N°001>
<CLT 4>Just some text to test my new regex !<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
<ORIGINAL N°002>
<CLT 4>This mailbox doesn’t seem important right now.<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
<ORIGINAL N°003>
<CLT 4>The entrance is still blocked by that giant hunk of metal.<CLT>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
<ORIGINAL N°004>
<CLT 4>An other text to see how the regex works<CLT>

Notes on the 2nd regex :

First, the in-line modifier (?-s) means that the regex char . represents a single standard character, only !
Then, the part (?:(?!<|>|-).)+? looks for the smallest non-null range of standard characters, different from <, > and -, in a non-capturing group, till… … … the next main part
This main part is \K(?:\h*\R\h*)++ which resets the regex engine first ( \K ), then searches for any non-null blocks of EOL chars, possibly surrounded with blank characters, placed in a non-capturing group, without any backtracking in that part, due to the possessive quantifier ++
The final part (?=(<CLT>)|.+) is a dummy look-ahead structure, which defines an always-true condition ! Indeed, the block of EOL chars searched must be followed by, either, the string <CLT>, stored as group 1 or any non-null range of standard chars
The main advantage of this odd syntax is that, if the EOL chars are followed with the specific string <CLT>, we won’t replace with a space character but simply delete this EOL block, which can be achieved with the conditional replacement ?1:\x20, meaning :
If group 1 exists, replace with the EMPTY string
If group 1 does not exist, replace with a space character

Best Regards,

guy038

Joey Flaig

@guy038

I hope you and your family had a good time! It’s been a long time, but my dad still misses going on field trips with me and my class when I was younger. Precious memories! :DD I’m glad he was with me when I fell in the fish pond at the local zoo!

Thank you for all of this! This is so close, aaaaaaahhhh! I just realized - this would’ve been a lot easier if I’d just linked one of the transcript files:
https://drive.google.com/open?id=141c3d3ZBOb_GG3Psb9eQaaiWUl-g-ATF

I followed what you said to do in your most recent post. Doing the first regex (to cut out the extra data I don’t need) works, but when I do the second regex (to turn the multi-line dialogue into mono-line dialogue), the problem from my previous post still happens:

https://drive.google.com/open?id=11ChQl7D9YZ80COaxyadjjBl-tIp5aTPJ

I tried doing the mono-line regex first. Then that one did its job and looked like your expected text… but when I did the extra data removal regex after, then that one didn’t work!

guy038

Hi, @@joey-flaig and All,

This time, no trouble ! I’ve found the right regex S/R, which is, BTW, much more simple that anything else tried before !

As usual, having the exact bunch of data, to work on, is always the best solution !

After downloading your complete XML file, I noticed two things :

Some dialog do not begin with any tag <CLT ##> nor <CLT> => We cannot rely on these tags and should prefer to consider <ORIGINAL N°###> as an anchor :-))
When dialog is multi-lines, it is always split in two lines !
More anecdotal, your file is an UNIX file, with the UCS-2 LE BOM encoding

So, as a summary :

To remove the extra data, not needed, use the A S/R :
- SEARCH (?s-i)^\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\1>\R
- REPLACE Leave EMPTY
To change the two-lines dialog in a one-line dialog, use the B S/R :
- SEARCH (?-si)<ORIGINAL N°\d+>\R.+\K\R(?!-{60}|</ORIGINAL N°\d+>)
- REPLACE \x20

IMPORTANT :

Assuming your present data, your may execute the search/replacement B, first and, secondly, the A S/R ;-))
I’ve supposed that your separation line always contains 60 dashes, exactly. If not, change the value 60 , between braces, in the search regex, accordingly
Remember to click on the Replace All button, exclusively, when running the B S/R !

NOTES on the B new regex :

First, the part <ORIGINAL N°\d+>\R.+ looks for the complete line <ORIGINAL N°###> with its EOL character, followed with all the standard characters of the next line ( 1st line of dialog )
Then, the syntax \K\R resets the regex engine and it, now, looks the the EOL characters of the 1st line of dialog
But this search occurred ONLY IF the EOL chars are NOT followed with, either, a line of 60 dashes OR the string </ORIGINAL N°###>

Best regards,

guy038

Joey Flaig

@guy038
I wish I’d thought to send you the files themselves at first, haha! You’re great!

The B S/R regex to turn two-line dialogues into one-line dialogues works like a charm!

However, the a S/R regex to remove the extra data doesn’t seem to work at all. I tried it in the order of B + A, then A + B, and even A by itself, but it’s not doing anything. I made sure to always use Replace All for both regexes. Is there a setting I shouldn’t have checked?
https://imgur.com/nIPe0w3

Also, if it matters - I’m using the 32 bit version of NPP [I had to in order to use the Combiner plugin]

guy038

Hello @@joey-flaig and All,

Aaaah ! As soon as I saw the picture of your text, in the background, I understood what happened ;-))

Of course, considering the first entry N°001 and supposing you’ve executed the B regex S/R first ( 46 occurrences replaced ), I was expecting :

<SPEAKER N°001>KIYOTAKA ISHIMARU</SPEAKER N°001>
<ORIGINAL N°001>
Would you like to study with me, Makoto? Just the two of us?
</ORIGINAL N°001>
<JAPANESE N°001>
苗木くん！
どうだ、これから一緒に自習しないか！？
</JAPANESE N°001>
<TRANSLATED N°001>

</TRANSLATED N°001>
<COMMENT N°001>

</COMMENT N°001>

But, seemingly, you’ve already changed the N°001 entry, as below :

<SPEAKER N°001>KIYOTAKA ISHIMARU</SPEAKER N°001>
<ORIGINAL N°001>
Would you like to study with me, Makoto? Just the two of us?</ORIGINAL N°001><JAPANESE N°001>
苗木くん！どうだ、これから一緒に自習しないか！？</JAPANESE N°001>
<TRANSLATED N°001>

</TRANSLATED N°001>
<COMMENT N°001>

</COMMENT N°001>

It easy to see why this new text layout breaks the logic of the A search regex. Indeed, I, initially, supposed that the part </ORIGINAL N°001> was, always, at beginning of lines !

So, the final S/R A becomes :

SEARCH (?s-i)(^)?\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\2>\R

REPLACE ?1:\r\n

Notes :

This new A regex looks for the string </ORIGINAL N°###>, possibly preceded with blank characters, at any location of current line, due to the optional syntax (^)?
In replacement, 2 possibilities :
- If the string </ORIGINAL N°###> is at a beginning of current line, the group 1 exists, so the conditional replacement ?1 rewrites all text before the : char, so … nothing
- If the string </ORIGINAL N°###> is located elsewhere, the ^ location is not true. So the group1 is not defined and the conditional replacement ?1 rewrites all text, after the : char, so a line break \r\n

That’s all ;-)) This time, you should see the message Replace All: 82 occurrences were replaced

Remark : Of course, I, also, verified that your new text layout does not break the logic of S/R B !

Cheers,

guy038

Joey Flaig

@guy038
It works! Thank you so much for all the help, and for accommodating my requests and frequent regex-breaking! It means a lot to me, and you’ve saved me soooo much time and hand pain!! Is there any way I can make it up to you? I could draw you something :D

guy038

Hi Joey,

Thanks for your kind words ! You said :

Is there any way I can make it up to you?

Thanks, but I don’t need anything !! I’m just pleased that the last regexes are the good ones for your specific file !

This is the main point, indeed ! Regexes are very, very, very sensitive to text layout. So, once I’ve built up some regex for an OP, based on his provided examples, the OP should not add, delete or modify anything of the original text, in the meanwhile, as, probably, the regex will not work anymore ;-))

As far as possible, anyone, asking for regex solutions, should consider all cases of text layouts, of the original file to modify ;-))

It’s generally, not so obvious, and, in my personal work, I simply create successive versions of the regex to get a final version which handles all reasonable cases !

I say reasonable ( and not possible ) because, sometimes, we can’t think about all the possibilities and, anyway, this could lead to an huge and useless regex ;-))

Best Regards

guy038

Finding and replacing all data between two points?

<SPEAKER N°004>MONOKUMA</SPEAKER N°004> <ORIGINAL N°004> Well then. Since you’re all giving it your best, your generous headmaster will give you a little hint!

<SPEAKER N°004>MONOKUMA</SPEAKER N°004>
<ORIGINAL N°004>
Well then. Since you’re all giving it your best, your
generous headmaster will give you a little hint!