Community
    • Login

    Finding and replacing all data between two points?

    Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
    26 Posts 4 Posters 14.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • guy038G
      guy038
      last edited by guy038

      Hi @@joey-flaig and All,

      Ah, I clearly see, now, what you’re looking for ;-)) Actually, you don’t mind about the tags <CLT 4> and <CLT> at all !

      So, the magic regex is :

      SEARCH (?-si)^<ORIGINAL N°\d+>\R.+?\K\h*\R\h*(?!<)

      REPLACE \x20

      • Tick, preferably, the Wrap around option

      • Select the Regular expression search mode

      • Due to the \K syntax, you must use the Replace All button, exclusively

      Et voilà !


      Notes :

      • First, as usual, the in-line modifiers (?s-i) :

        • Forces to regex engine to interpret any dot ( . ) as representing any single character, even EOL characters, (?-s)

        • Forces the search to be processed in a non-insensitive way, (?-i)

      • Then, from beginning of line ( ^ ), the regex searches for the string <ORIGINAL N°, followed with some digits ( \d+ ), followed with the > symbol, closed to its EOL chars ( \R)

      • Now, on the line, following the line <ORIGINAL N°###>, it looks for the shortest range of standard characters ( .+? )

      • At this point, the \K syntax resets the regex engine search and position. So, from now on, the final search looks for possible blank characters followed with EOL chars, followed, again, with possible blank chars ( \h*\R\h* )

      • But this search happens ONLY IF  the negative look-ahead (?!<) is verified, i.e. if the next line does not begin with the < symbol ( => is, simply, the continuation text of the previous line )

      • In that case, the EOL characters, possibly surrounded with blank characters, are replaced with a single space character ( \x20 )

      Best Regards,

      guy038

      P.S. :

      • The fact of searching for \h*\R\h* and replacing the \x20 ensures you that the two parts of the text will be always joined with an unique space character ;-))

      • If, between the tags <ORIGINAL N°###> and </ORIGINAL N°###>, you have more than 2 lines of text, like below :

      <ORIGINAL N°001>
      <CLT 4>The entrance
          is still blocked
              by that
      giant
         hunk of
      metal.<CLT>
      </ORIGINAL N°001>
      

      No problem ! Just click, several times, on the Replace All button, until the message Replace All: 0 occurrences were replaced. occurs ;-))

      And you’ll be left with :

      <ORIGINAL N°001>
      <CLT 4>The entrance is still blocked by that giant hunk of metal.<CLT>
      </ORIGINAL N°001>
      
      1 Reply Last reply Reply Quote 3
      • Joey FlaigJ
        Joey Flaig
        last edited by

        Yeah, this is what I was looking to do! This regex looks great… but it sometimes fuses the line indicating a speaker change [the ones with all the dashes that look like ----------------) with the dialogue! Any way to tweak the regex to prevent that?

        https://imgur.com/OhYLhBM

        If it matters, the speaker change lines contain 60 dashes!

        1 Reply Last reply Reply Quote 0
        • guy038G
          guy038
          last edited by

          Hi, @@joey-flaig and All,

          I’ve got no much time, right now ! So try the exact regex, below :

          SEARCH (?-si)^<ORIGINAL N°\d+>\R.+?\K\h*\R\h*+(?!<|-{5,})

          REPLACE \x20

          I’ll explain to you when I’m back, about six hours later !

          Cheers,

          guy038

          Alan KilbornA 1 Reply Last reply Reply Quote 3
          • Alan KilbornA
            Alan Kilborn @guy038
            last edited by

            @guy038 said:

            I’ve got no much time, right now

            And it seems like you could have a full time job, writing this person’s regexes!

            1 Reply Last reply Reply Quote 4
            • guy038G
              guy038
              last edited by guy038

              Hello, @@joey-flaig and All,

              So yesterday, my wife and I were accompanying 7/8 years old children ( our daughter’s class +2 other classes ), on a school trip and then, in the late afternoon, I preferred to watch the USA-Spain match of the women’s world football championship ! Of course, I wanted to know who France’s opponent would be ;-))


              Ah ! I understood why the separation line, made up of 60 dashes, was missing in the original Joey’s post ! Just because, due to the MarkDown syntax, any line of, at least, 3 dashes produces a slight gray horizontal rule, instead ;-))

              So, Joey, just forget, my quick previous post and let’s recapitulate all the process :


              We’ll start with this sample text of 4 blocks, between the lines of dashes. In two of them, I placed some multi-lines text, inside the section <CLT 4>.............<CLT>

              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
              <ORIGINAL N°001>
              <CLT 4>Just some   
                  
                 
                   text to test
              my new
              regex !   
              <CLT>
              </ORIGINAL N°001>
              <JAPANESE N°001>
              <CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
              </JAPANESE N°001>
              <TRANSLATED N°001>
              
              </TRANSLATED N°001>
              <COMMENT N°001>
              </COMMENT N°001>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
              <ORIGINAL N°002>
              <CLT 4>This mailbox doesn’t seem important right now.<CLT>
              </ORIGINAL N°002>
              <JAPANESE N°002>
              <CLT 4>このレターケースは…
              今回の事件には関係ないよな…<CLT>
              </JAPANESE N°002>
              <TRANSLATED N°002>
              
              </TRANSLATED N°002>
              <COMMENT N°002>
              </COMMENT N°002>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
              <ORIGINAL N°003>
              <CLT 4>The entrance is still blocked by that giant hunk of
              metal.<CLT>
              </ORIGINAL N°003>
              <JAPANESE N°003>
              <CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
              </JAPANESE N°003>
              <TRANSLATED N°003>
              
              </TRANSLATED N°003>
              <COMMENT N°003>
              </COMMENT N°003>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
              <ORIGINAL N°004>
              <CLT 4>An other   
                   text to see
              how the
              regex works<CLT>
              </ORIGINAL N°004>
              <JAPANESE N°004>
              <CLT 4>入口は…鉄の塊で塞がれてしまっている…<CLT>
              </JAPANESE N°004>
              <TRANSLATED N°004>
              
              </TRANSLATED N°004>
              <COMMENT N°004>
              </COMMENT N°004>
              

              Using the regex S/R, below, we keep the interesting part of each block, from Joey’s point of view, of course !

              SEARCH (?s-i)^\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\1>\R

              REPLACE Leave EMPTY

              And we get :

              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
              <ORIGINAL N°001>
              <CLT 4>Just some   
                  
                 
                   text to test
              my new
              regex !   
              <CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
              <ORIGINAL N°002>
              <CLT 4>This mailbox doesn’t seem important right now.<CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
              <ORIGINAL N°003>
              <CLT 4>The entrance is still blocked by that giant hunk of
              metal.<CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
              <ORIGINAL N°004>
              <CLT 4>An other   
                   text to see
              how the
              regex works<CLT>
              

              Now, in order to change all multi-lines sections <CLT 4>.............<CLT> into a mono-line one, use the following regex S/R :

              SEARCH (?-s)(?:(?!<|>|-).)+?\K(?:\h*\R\h*)++(?=(<CLT>)|.+)

              REPLACE ?1:\x20

              And here is your expected text :

              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°001>MAKOTO NAEGI</SPEAKER N°001>
              <ORIGINAL N°001>
              <CLT 4>Just some text to test my new regex !<CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°002>MAKOTO NAEGI</SPEAKER N°002>
              <ORIGINAL N°002>
              <CLT 4>This mailbox doesn’t seem important right now.<CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°003>MAKOTO NAEGI</SPEAKER N°003>
              <ORIGINAL N°003>
              <CLT 4>The entrance is still blocked by that giant hunk of metal.<CLT>
              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
              <SPEAKER N°004>MAKOTO NAEGI</SPEAKER N°004>
              <ORIGINAL N°004>
              <CLT 4>An other text to see how the regex works<CLT>
              

              Notes on the 2nd regex :

              • First, the in-line modifier (?-s) means that the regex char . represents a single standard character, only !

              • Then, the part (?:(?!<|>|-).)+? looks for the smallest non-null range of standard characters, different from <, > and -, in a non-capturing group, till… … … the next main part

              • This main part is \K(?:\h*\R\h*)++ which resets the regex engine first ( \K ), then searches for any non-null blocks of EOL chars, possibly surrounded with blank characters, placed in a non-capturing group, without any backtracking in that part, due to the possessive quantifier ++

              • The final part (?=(<CLT>)|.+) is a dummy look-ahead structure, which defines an always-true condition ! Indeed, the block of EOL chars searched must be followed by, either, the string <CLT>, stored as group 1 or any non-null range of standard chars

              • The main advantage of this odd syntax is that, if the EOL chars are followed with the specific string <CLT>, we won’t replace with a space character but simply delete this EOL block, which can be achieved with the conditional replacement ?1:\x20, meaning :

              • If group 1 exists, replace with the EMPTY string

              • If group 1 does not exist, replace with a space character

              Best Regards,

              guy038

              1 Reply Last reply Reply Quote 3
              • Joey FlaigJ
                Joey Flaig
                last edited by Joey Flaig

                @guy038

                I hope you and your family had a good time! It’s been a long time, but my dad still misses going on field trips with me and my class when I was younger. Precious memories! :DD I’m glad he was with me when I fell in the fish pond at the local zoo!

                Thank you for all of this! This is so close, aaaaaaahhhh! I just realized - this would’ve been a lot easier if I’d just linked one of the transcript files:
                https://drive.google.com/open?id=141c3d3ZBOb_GG3Psb9eQaaiWUl-g-ATF

                I followed what you said to do in your most recent post. Doing the first regex (to cut out the extra data I don’t need) works, but when I do the second regex (to turn the multi-line dialogue into mono-line dialogue), the problem from my previous post still happens:

                https://drive.google.com/open?id=11ChQl7D9YZ80COaxyadjjBl-tIp5aTPJ

                I tried doing the mono-line regex first. Then that one did its job and looked like your expected text… but when I did the extra data removal regex after, then that one didn’t work!

                1 Reply Last reply Reply Quote 0
                • guy038G
                  guy038
                  last edited by guy038

                  Hi, @@joey-flaig and All,

                  This time, no trouble ! I’ve found the right regex S/R, which is, BTW, much more simple that anything else tried before !

                  As usual, having the exact bunch of data, to work on, is always the best solution !

                  After downloading your complete XML file, I noticed two things :

                  • Some dialog do not begin with any tag <CLT ##> nor <CLT> => We cannot rely on these tags and should prefer to consider <ORIGINAL N°###> as an anchor :-))

                  • When dialog is multi-lines, it is always split in two lines !

                  • More anecdotal, your file is an UNIX file, with the UCS-2 LE BOM encoding


                  So, as a summary :

                  • To remove the extra data, not needed, use the A S/R :

                    • SEARCH (?s-i)^\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\1>\R

                    • REPLACE Leave EMPTY

                  • To change the two-lines dialog in a one-line dialog, use the B S/R :

                    • SEARCH (?-si)<ORIGINAL N°\d+>\R.+\K\R(?!-{60}|</ORIGINAL N°\d+>)

                    • REPLACE \x20


                  IMPORTANT :

                  • Assuming your present data, your may execute the search/replacement B, first and, secondly, the A S/R ;-))

                  • I’ve supposed that your separation line always contains 60 dashes, exactly. If not, change the value 60 , between braces, in the search regex, accordingly

                  • Remember to click on the Replace All button, exclusively, when running the B S/R !


                  NOTES on the B new regex :

                  • First, the part <ORIGINAL N°\d+>\R.+ looks for the complete line <ORIGINAL N°###> with its EOL character, followed with all the standard characters of the next line ( 1st line of dialog )

                  • Then, the syntax \K\R resets the regex engine and it, now, looks the the EOL characters of the 1st line of dialog

                  • But this search occurred ONLY IF  the EOL chars are NOT followed with, either, a line of 60 dashes OR the string </ORIGINAL N°###>

                  Best regards,

                  guy038

                  1 Reply Last reply Reply Quote 2
                  • Joey FlaigJ
                    Joey Flaig
                    last edited by Joey Flaig

                    @guy038
                    I wish I’d thought to send you the files themselves at first, haha! You’re great!

                    The B S/R regex to turn two-line dialogues into one-line dialogues works like a charm!

                    However, the a S/R regex to remove the extra data doesn’t seem to work at all. I tried it in the order of B + A, then A + B, and even A by itself, but it’s not doing anything. I made sure to always use Replace All for both regexes. Is there a setting I shouldn’t have checked?
                    https://imgur.com/nIPe0w3

                    Also, if it matters - I’m using the 32 bit version of NPP [I had to in order to use the Combiner plugin]

                    1 Reply Last reply Reply Quote 0
                    • guy038G
                      guy038
                      last edited by guy038

                      Hello @@joey-flaig and All,

                      Aaaah ! As soon as I saw the picture of your text, in the background, I understood what happened ;-))

                      Of course, considering the first entry N°001 and supposing you’ve executed the B regex S/R first ( 46 occurrences replaced ), I was expecting :

                      <SPEAKER N°001>KIYOTAKA ISHIMARU</SPEAKER N°001>
                      <ORIGINAL N°001>
                      Would you like to study with me, Makoto? Just the two of us?
                      </ORIGINAL N°001>
                      <JAPANESE N°001>
                      苗木くん!
                      どうだ、これから一緒に自習しないか!?
                      </JAPANESE N°001>
                      <TRANSLATED N°001>
                      
                      </TRANSLATED N°001>
                      <COMMENT N°001>
                      
                      </COMMENT N°001>
                      

                      But, seemingly, you’ve already changed the N°001 entry, as below :

                      <SPEAKER N°001>KIYOTAKA ISHIMARU</SPEAKER N°001>
                      <ORIGINAL N°001>
                      Would you like to study with me, Makoto? Just the two of us?</ORIGINAL N°001><JAPANESE N°001>
                      苗木くん!どうだ、これから一緒に自習しないか!?</JAPANESE N°001>
                      <TRANSLATED N°001>
                      
                      </TRANSLATED N°001>
                      <COMMENT N°001>
                      
                      </COMMENT N°001>
                      

                      It easy to see why this new text layout breaks the logic of the A search regex. Indeed, I, initially, supposed that the part </ORIGINAL N°001> was, always, at beginning of lines !

                      So, the final S/R A becomes :

                      SEARCH (?s-i)(^)?\h*</ORIGINAL\x20N°(\d+)>.+?</COMMENT\x20N°\2>\R

                      REPLACE ?1:\r\n

                      Notes :

                      • This new A regex looks for the string </ORIGINAL N°###>, possibly preceded with blank characters, at any location of current line, due to the optional syntax (^)?

                      • In replacement, 2 possibilities :

                        • If the string </ORIGINAL N°###> is at a beginning of current line, the group 1 exists, so the conditional replacement ?1 rewrites all text before the : char, so … nothing

                        • If the string </ORIGINAL N°###> is located elsewhere, the ^ location is not true. So the group1 is not defined and the conditional replacement ?1 rewrites all text, after the : char, so a line break \r\n

                      That’s all ;-)) This time, you should see the message Replace All: 82 occurrences were replaced

                      Remark : Of course, I, also, verified that your new text layout does not break the logic of S/R B !

                      Cheers,

                      guy038

                      1 Reply Last reply Reply Quote 1
                      • Joey FlaigJ
                        Joey Flaig
                        last edited by

                        @guy038
                        It works! Thank you so much for all the help, and for accommodating my requests and frequent regex-breaking! It means a lot to me, and you’ve saved me soooo much time and hand pain!! Is there any way I can make it up to you? I could draw you something :D

                        1 Reply Last reply Reply Quote 1
                        • guy038G
                          guy038
                          last edited by guy038

                          Hi Joey,

                          Thanks for your kind words ! You said :

                          Is there any way I can make it up to you?

                          Thanks, but I don’t need anything !! I’m just pleased that the last regexes are the good ones for your specific file !


                          This is the main point, indeed ! Regexes are very, very, very sensitive to text layout. So, once I’ve built up some regex for an OP, based on his provided examples, the OP should not add, delete or modify anything of the original text, in the meanwhile, as, probably, the regex will not work anymore ;-))

                          As far as possible, anyone, asking for regex solutions, should consider all cases of text layouts, of the original file to modify ;-))

                          It’s generally, not so obvious, and, in my personal work, I simply create successive versions of the regex to get a final version which handles all reasonable cases !

                          I say reasonable ( and not possible ) because, sometimes, we can’t think about all the possibilities and, anyway, this could lead to an huge and useless regex ;-))

                          Best Regards

                          guy038

                          1 Reply Last reply Reply Quote 1
                          • First post
                            Last post
                          The Community of users of the Notepad++ text editor.
                          Powered by NodeBB | Contributors