• Login
Community
  • Login

sort file removing duplicates possible?

Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
75 Posts 5 Posters 44.8k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • P
    patrickdrd
    last edited by Jul 12, 2017, 7:58 AM

    one of the functions I was using in my previous editor was
    “sort file removing duplicates”,
    is this possible using np++?

    S 1 Reply Last reply Jul 12, 2017, 12:09 PM Reply Quote 0
    • S
      Scott Sumner @patrickdrd
      last edited by Jul 12, 2017, 12:09 PM

      @patrickdrd

      A direct command for this does not exist (natively), but you can record and save a macro to do it!

      Here’s how:

      • Select Macro (menu) -> Start Recording
      • Select Edit (menu) -> Line Operations -> Sort Lines Lexicographically Ascending
      • Press Ctrl+h (to invoke the replace dialog box).
      • In the Find what field, type (?-s)^(.*)(?:\R)(?s)(?=.*^\1\R)
      • Clear out anything that appears in the Replace with field; make it empty!
      • Set the Search mode to Regular expression.
      • Uncheck all other checkboxes in the Replace dialog.
      • Press the Replace all button.
      • Select Macro (menu) -> Stop Recording
      • Select Macro (menu) -> Save Currently Recorded Macro; give it a good name, maybe assign a keyboard shortcut.

      To run it, either use the keyboard shortcut (if you assigned one), or use the Macro menu and select the menu entry for the name you gave the macro.

      For the technical, detail-oriented reader, what happens internally is that a block like this is inserted into the shortcuts.xml file:

          <Macro name="test sort and del dupe lines" Ctrl="no" Alt="no" Shift="no" Key="0">
              <Action type="2" message="0" wParam="42059" lParam="0" sParam="" />
              <Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
              <Action type="3" message="1601" wParam="0" lParam="0" sParam="(?-s)^(.*)(?:\R)(?s)(?=.*^\1\R)" />
              <Action type="3" message="1625" wParam="0" lParam="2" sParam="" />
              <Action type="3" message="1602" wParam="0" lParam="0" sParam="" />
              <Action type="3" message="1702" wParam="0" lParam="512" sParam="" />
              <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
          </Macro>
      

      If this (or ANY) posting was useful, don’t post a “thanks”, just upvote ( click the ^ in the ^ 0 v area on the right ).

      S 1 Reply Last reply Jul 13, 2017, 5:53 PM Reply Quote 0
      • S
        Scott Sumner @Scott Sumner
        last edited by Jul 13, 2017, 5:53 PM

        The above macro operates on the entire file in the active editor window when it is run. Perhaps even more useful is to have it operate only on the lines making up a selection. This is an easy modification. Simply select/highlight a block of text before starting to record the macro. This will cause the In selection checkbox on the Replace tab of the Find dialog to be enabled. Check this checkbox before running the Replace All (while recording the macro).

        Equivalently, edit shortcuts.xml and change the message=“1702” line to contain lParam=“640” instead of 512. This adds in the check in the In selection box to the already-present Direction: Down setting for the replacement. The various meanings for the 1702 value, as well as the encodings for all of the lines in the macro section of shortcuts.xml may be found here: http://docs.notepad-plus-plus.org/index.php/Editing_Configuration_Files#Search_.2F_Replace_encoding Note that while researching this I discovered that this documentation appears to be incorrect for the search direction–it seems that downward appears to be 512, not upwards. Side note: 640 = 512 + 128, meaning that both a downward search and an in-selection search are in effect when the replacement runs within the macro.

        Note that if this “selection” change to the macro is implemented, it is necessary to select beforehand the entire file (easy way: ctrl+a), if the action is desired upon the entire file contents. This is a change from the first version of the macro, where it acted upon the whole file by default and nothing needed to be selected.

        Also, be careful when specifying the range of lines in your selection. It is very easy to get one more line than intended at the tail end of your range, if you do the natural thing (with the mouse, at least) and select from a point earlier in the file to a later point. I find there is less confusion about this if I remember to select from a later point (greater line number) to an earlier point (lesser line number).

        As an example, if your intended sort selection in the following is lines ddd through bbb, and you select like this:

        you will pull (undesired) line aaa into the sort as well (the fact that you can see the thin caret is on the aaa line is the key) and the sort will result in the undesired:

        However, selecting from the bottom up (start on bbb and making the selection cover ddd) makes it clearer to see what will be sorted:

        The reality of the way it works is that any line with at least a partial selection will be included, so even this works just as well:

        The most recent two selections shown yield the following sort result, as desired:

        1 Reply Last reply Reply Quote 2
        • P
          patrickdrd
          last edited by Oct 5, 2017, 9:50 AM

          something is going wrong:
          first of all, I’m using:
          <Action type=“3” message=“1702” wParam=“0” lParam=“768” sParam=“” />

          in order to do the command for the whole file,
          but I notice the last line is duplicate after the command finishes!

          even if I run it again, same result,
          if I run a find though, the last line is found 2 times!

          1 Reply Last reply Reply Quote 0
          • P
            patrickdrd
            last edited by Oct 5, 2017, 10:10 AM

            this should do it (the problem was in the removing of duplicates):
            ^(.?)$\s+?^(?=.^\1$)

            1 Reply Last reply Reply Quote 0
            • S
              Scott Sumner
              last edited by Scott Sumner Oct 5, 2017, 1:17 PM Oct 5, 2017, 1:17 PM

              @patrickdrd

              Your regular expression will not work as shown; I can tell some of the important characters in it are missing (due to being consumed by the Markdown syntax this website uses). If you need to write a * here, either escape it, like this: \* in which it will then appear like this: *…or put it inside backticks (also called grave accents), like this `*` in which case it will then appear like this: * (I find this second way of doing it easier/cleaner).

              So let’s assume you meant to post ^(.*?)$\s+?^(?=.*^\1$)

              That is clearly wrong as it introduces the notion of whitespace (with the \s+). That may work for exactly what YOU are doing, but in general; if we are sorting lines and then deleting duplicates then \s doesn’t belong.

              I retested the original test sort and del dupe lines macro definition I posted earlier, and it (still) works. So let’s try an example, here’s some random unsorted data with some duplicates:

              nonprehensile
              aeriferous
              calabrian
              estimating
              urethrostomy
              redigging
              estimating
              crystalling
              cynwulf
              fetor
              kali
              armada
              auspicate
              tusser
              fetor
              zoophyte
              ribless
              fetor
              deducibly
              calabrian
              estimating
              expressivity
              

              After running the macro, we obtain the desired result (sorted, duplicates removed), nothing more, nothing less:

              aeriferous
              armada
              auspicate
              calabrian
              crystalling
              cynwulf
              deducibly
              estimating
              expressivity
              fetor
              kali
              nonprehensile
              redigging
              ribless
              tusser
              urethrostomy
              zoophyte
              

              Earlier I presented the regular expression (in the macro) without explanation. Here’s what it is actually doing to find matches:

              (?-s)   ^   (.*)   (?:   \R   )   (?s)   (?=   .*   ^   \1   \R   )
              ^^^^^-----------------------------------------------------------------------------> after this . appearing will match any single character except a line-ending character
                      ^-------------------------------------------------------------------------> match only at start of line
                          ^  ^------------------------------------------------------------------> remember what matches the expression between `(` and `)`; call it "group #1"
                           ^^-------------------------------------------------------------------> match zero or more characters (in this case it will stop matching at the next line-ending character)
                                 ^^^        ^---------------------------------------------------> the syntax `(?:` followed by something followed by `)` means to match what is inside but don't remember it
                                       ^^-------------------------------------------------------> match a line-ending
                                                ^^^^--------------------------------------------> after this a . appearing will match any single character including a line-ending character
                                                       ^^^                      ^---------------> the syntax `(?=` followed by something followed by `)` means whatever is inside needs to match for the overall expression to match, but it isn't consumed and isn't part of the match
                                                             ^^---------------------------------> match zero or more characters (ANY characters, including line-ending characters...this will match across lines)
                                                                  ^-----------------------------> match at start of line
                                                                      ^^------------------------> match what we remembered earlier as "group #1"
                                                                           ^^-------------------> match a line-ending character
              

              So that’s actually still pretty technical/confusing. Maybe this is better:

              • look at the complete contents of the current line
              • glance further down (any arbitrary amount of distance) in the file to see if there is an exact duplicate of the current line contents; if so, the current line is a MATCH
              • if there was a MATCH, do the REPLACEMENT (see below)
              • move beyond the current line (go down one line)
              • repeat these steps until the file is exhausted

              When there is a MATCH, perform the specified REPLACEMENT on the matched text. Since the matched text is the entire current line (with line-ending), and the replacement is NOTHING, this effectively deletes the current line. This is what is desired because our goal is to removed duplicate lines and we’ve already established that the matched line occurs again further down in the file.

              Dedicated regexers might catch that my regular expression is more-complicated than needed. The (?:\R) part could be changed to simply \R. (?: ) is normally used when some grouping without remembering is necessary. Here \R needs no grouping so the (?: ) is just extra fluff…does no harm except making the whole expression harder to read.

              Super-dedicated regexers would also say that the regular expression is more-complicated than needed because at the point it is run (inside the macro above) the file is already sorted! Very true. The regular expression above works for sorted or unsorted duplicate line detection/removal. Here’s the simpler one that will only work on the sorted lines situation: (?-s)^(.*)\R(?=^\1\R)

              One more thing worth mentioning is that if you want to delete duplicate lines but ignore/leave blank/(empty!) lines, simply change (.*) in the regular expression to (.+). Instead of “match ZERO or more characters”, this change will “match ONE or more characters”, effectively failing to match truly empty lines so they won’t be removed.

              1 Reply Last reply Reply Quote 2
              • P
                patrickdrd
                last edited by Oct 5, 2017, 1:33 PM

                I had problem with a file today, I don’t know why,
                is there any way I can send you the file to take a look?

                S 1 Reply Last reply Oct 5, 2017, 1:38 PM Reply Quote 0
                • S
                  Scott Sumner @patrickdrd
                  last edited by Oct 5, 2017, 1:38 PM

                  @patrickdrd said:

                  send you the file to take a look

                  Post its contents on http://textuploader.com/ and reply here with the link to it. Also indicate exactly what problem you are seeing if it isn’t obvious. No guarantees about how deeply I can get involved, but I’ll take a quick look… :-)

                  1 Reply Last reply Reply Quote 0
                  • P
                    patrickdrd
                    last edited by Oct 5, 2017, 1:55 PM

                    ok, it’s here:
                    http://textuploader.com/d4dkr

                    removing duplicates results contain a duplicate result

                    1 Reply Last reply Reply Quote 0
                    • P
                      patrickdrd
                      last edited by patrickdrd Oct 5, 2017, 2:35 PM Oct 5, 2017, 2:34 PM

                      remove duplicates without sorting produces the result:
                      DBServerIP=10.1.249.215
                      DBServerIP=10.12.77.185
                      DBServerIP=10.1.249.215

                      (1st and 3rd exact duplicates)

                      and sorting first and then removing dups results in:
                      DBServerIP=10.12.22.129
                      DBServerIP=10.12.77.185
                      DBServerIP=10.12.77.185

                      last 2 duplicated

                      (using your regular expression in both cases of course)

                      S 2 Replies Last reply Oct 5, 2017, 2:43 PM Reply Quote 0
                      • S
                        Scott Sumner @patrickdrd
                        last edited by Oct 5, 2017, 2:43 PM

                        @patrickdrd

                        Okay, so a quick look told me what is happening. It’s interesting. :-)

                        If you run only the sort on this data, you’ll get this at the end of the file:

                        Imgur

                        You’d think that the removal of duplicate lines after this would result in only a single occurrence of DBServerIP=10.12.77.185. However, if we look closer at what is really left after the duplicate removal (turning on visibility of line-endings!), we see:

                        Imgur

                        We see that the two lines with that IP address truly are different–one has a line-ending and one doesn’t–and because these two lines are not the same, the regular-expression replacement is working correctly by leaving both of these lines after it does its work.

                        All of the original lines that ended in .185 had line-endings (in other words, were exactly the same), so I’d say this is an artifact resulting from the sort operation (in my mind this is a sort BUG!).

                        But we can work around it. There could be a regular expression solution, but maybe the regex that removes duplicates is complicated enough. What I’d suggest here is to modify the original macro, after the sort but before the find+replace, to:

                        • move caret position to the end of file (the sort operation leaves it at the beginning of file)
                        • insert a (Windows style) line-ending
                        • move caret position back to beginning of file (in preparation for the find+replace)

                        Thus:

                        <Macro name="test sort and del dupe lines 2" Ctrl="no" Alt="no" Shift="no" Key="0">
                            <Action type="2" message="0" wParam="42059" lParam="0" sParam="" />
                            <Action type="0" message="2318" wParam="0" lParam="0" sParam="" />
                            <Action type="1" message="2170" wParam="0" lParam="0" sParam="&#x000D;" />
                            <Action type="1" message="2170" wParam="0" lParam="0" sParam="&#x000A;" />
                            <Action type="0" message="2316" wParam="0" lParam="0" sParam="" />
                            <Action type="3" message="1700" wParam="0" lParam="0" sParam="" />
                            <Action type="3" message="1601" wParam="0" lParam="0" sParam="(?-s)^(.*)(?:\R)(?s)(?=.*^\1\R)" />
                            <Action type="3" message="1625" wParam="0" lParam="2" sParam="" />
                            <Action type="3" message="1602" wParam="0" lParam="0" sParam="" />
                            <Action type="3" message="1702" wParam="0" lParam="512" sParam="" />
                            <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" />
                        </Macro>
                        

                        Compare that with the earlier version much earlier in this thread.

                        Running this new macro on your data results in:

                        Imgur

                        which is the desired result.

                        1 Reply Last reply Reply Quote 1
                        • S
                          Scott Sumner @patrickdrd
                          last edited by Oct 5, 2017, 2:58 PM

                          @patrickdrd

                          For the DBServerIP=10.1.249.215 case you mentioned, if you don’t do the sort first, then a .215 line is the last line of the data you turn over to the regular expression replace operation. Thus it lacks the trailing line-ending that the earlier occurrence of a similar line has. Same issue as the .185 case…

                          1 Reply Last reply Reply Quote 0
                          • P
                            patrickdrd
                            last edited by Oct 5, 2017, 4:04 PM

                            thanks a ton!

                            I’d a like a more “generic” approach, so I called trim 42056 to clear empty lines first,
                            because sorting puts an empty line at the top if is finds one,
                            then going on as you suggested

                            S 1 Reply Last reply Oct 5, 2017, 5:09 PM Reply Quote 0
                            • S
                              Scott Sumner @patrickdrd
                              last edited by Scott Sumner Oct 5, 2017, 5:10 PM Oct 5, 2017, 5:09 PM

                              @patrickdrd

                              I’m glad you have a solution. Yeah, the empty line thing with sorting is rather bad, but the end-user can’t control this behavior of the sorting. I think I’ve changed my mind and it is probably best to alter the regular expression a bit in order to handle the situation where there is a duplicate-but-without-line-ending at the end of the file. So I’d suggest changing it to:

                              (?-s)^(.*)\R(?s)(?=.*^\1(?:\R|\z))

                              I’ve done two things to this regex:

                              • I removed the (?: and ) around the first \R (a simplification discussed earlier so no need to say any more here)
                              • The final \R was changed to (?:\R|\z) (see discussion below)

                              The \R|\z part is what allows an almost-duplicate at the end-of-file-without-line-ending to be detected. The new part to this is the \z which roughly means “match only at the very end of the data”.

                              The (?: and ) was added so that the | only affects the \R that precedes it and the \z that follows it

                              1 Reply Last reply Reply Quote 2
                              • guy038G
                                guy038
                                last edited by Oct 6, 2017, 7:32 PM

                                Hello @patrickdrd, @scott-sumner and All,

                                Generally speaking, when you want to remove duplicate lines, from a PREVIOUSLY SORTED list, just use this simple regex S/R, below :

                                SEARCH (?-s)(.*\R)\1+

                                REPLACE \1

                                This regex is quite fast, because, in case of numerous duplicates, the part \1+ grabs all the duplicates ( with their EOL characters ), at once and just rewrites the first item of each block :-))

                                IMPORTANT : the last item of your sorted list must be followed by EOL character(s) !

                                Cheers,

                                guy038

                                1 Reply Last reply Reply Quote 3
                                • P
                                  patrickdrd
                                  last edited by May 31, 2018, 10:40 AM

                                  after trying both the macro and the textfx solution for a long time,
                                  I’ve seen that still ultraedit’s sorting works much better than both of them,
                                  I first rejected the macro for textfx’s favor but I found out lately that neither the latter does a good job, at least I prefer the sorting done by ultraedit ,sorry,
                                  I don’t have an example at the moment, I’ll post again when I do

                                  Claudia FrankC 2 Replies Last reply May 31, 2018, 12:48 PM Reply Quote 0
                                  • Claudia FrankC
                                    Claudia Frank @patrickdrd
                                    last edited by May 31, 2018, 12:48 PM

                                    @patrickdrd

                                    sorry, did not read the whole thread but what about a python one liner?

                                    editor.setText('\r\n'.join(list(set(editor.getText().splitlines()))))
                                    

                                    Cheers
                                    Claudia

                                    1 Reply Last reply Reply Quote 0
                                    • Claudia FrankC
                                      Claudia Frank @patrickdrd
                                      last edited by May 31, 2018, 12:52 PM

                                      @patrickdrd

                                      forgot sorting

                                      editor.setText('\r\n'.join(sorted(list(set(editor.getText().splitlines())))))
                                      

                                      Cheers
                                      Claudia

                                      1 Reply Last reply Reply Quote 1
                                      • P
                                        patrickdrd
                                        last edited by May 31, 2018, 1:05 PM

                                        thanks, I’m keeping that too and I’ll let you know

                                        1 Reply Last reply Reply Quote 0
                                        • P
                                          patrickdrd
                                          last edited by patrickdrd May 31, 2018, 1:23 PM May 31, 2018, 1:22 PM

                                          ok, I tested, this doesn’t work either somehow, at least doesn’t work like ue’s one, i.e.

                                          I tested with easylist from: https://easylist.to/easylist/easylist.txt (69883 lines),
                                          ue’s function cuts it to: 69238

                                          npp results:
                                          python scipt: 69818
                                          macro (scott) is a bit slow and results in 19610 lines (!) and
                                          textfx results in 69818 as well

                                          guy038 regular expression results in 28109 lines,
                                          according to my experience, I bet that ue’s result is the correct one,
                                          at least to my taste

                                          Claudia FrankC 1 Reply Last reply May 31, 2018, 1:24 PM Reply Quote 0
                                          • First post
                                            Last post
                                          The Community of users of the Notepad++ text editor.
                                          Powered by NodeBB | Contributors