Community
    • Login

    sort file removing duplicates possible?

    Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
    75 Posts 5 Posters 45.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • guy038G
      guy038
      last edited by

      Hello @patrickdrd, @scott-sumner and All,

      Generally speaking, when you want to remove duplicate lines, from a PREVIOUSLY SORTED list, just use this simple regex S/R, below :

      SEARCH (?-s)(.*\R)\1+

      REPLACE \1

      This regex is quite fast, because, in case of numerous duplicates, the part \1+ grabs all the duplicates ( with their EOL characters ), at once and just rewrites the first item of each block :-))

      IMPORTANT : the last item of your sorted list must be followed by EOL character(s) !

      Cheers,

      guy038

      1 Reply Last reply Reply Quote 3
      • patrickdrdP
        patrickdrd
        last edited by

        after trying both the macro and the textfx solution for a long time,
        I’ve seen that still ultraedit’s sorting works much better than both of them,
        I first rejected the macro for textfx’s favor but I found out lately that neither the latter does a good job, at least I prefer the sorting done by ultraedit ,sorry,
        I don’t have an example at the moment, I’ll post again when I do

        Claudia FrankC 2 Replies Last reply Reply Quote 0
        • Claudia FrankC
          Claudia Frank @patrickdrd
          last edited by

          @patrickdrd

          sorry, did not read the whole thread but what about a python one liner?

          editor.setText('\r\n'.join(list(set(editor.getText().splitlines()))))
          

          Cheers
          Claudia

          1 Reply Last reply Reply Quote 0
          • Claudia FrankC
            Claudia Frank @patrickdrd
            last edited by

            @patrickdrd

            forgot sorting

            editor.setText('\r\n'.join(sorted(list(set(editor.getText().splitlines())))))
            

            Cheers
            Claudia

            1 Reply Last reply Reply Quote 1
            • patrickdrdP
              patrickdrd
              last edited by

              thanks, I’m keeping that too and I’ll let you know

              1 Reply Last reply Reply Quote 0
              • patrickdrdP
                patrickdrd
                last edited by patrickdrd

                ok, I tested, this doesn’t work either somehow, at least doesn’t work like ue’s one, i.e.

                I tested with easylist from: https://easylist.to/easylist/easylist.txt (69883 lines),
                ue’s function cuts it to: 69238

                npp results:
                python scipt: 69818
                macro (scott) is a bit slow and results in 19610 lines (!) and
                textfx results in 69818 as well

                guy038 regular expression results in 28109 lines,
                according to my experience, I bet that ue’s result is the correct one,
                at least to my taste

                Claudia FrankC 1 Reply Last reply Reply Quote 0
                • Claudia FrankC
                  Claudia Frank @patrickdrd
                  last edited by

                  @patrickdrd

                  could you, by any chance, upload the ue cutted list?
                  To see the differences.

                  Cheers
                  Claudia

                  1 Reply Last reply Reply Quote 0
                  • patrickdrdP
                    patrickdrd
                    last edited by

                    yes, of course, please tell me where, pastebin doesn’t work, it’s blocked here (at work),
                    any other suggestions?

                    1 Reply Last reply Reply Quote 0
                    • Claudia FrankC
                      Claudia Frank
                      last edited by Claudia Frank

                      actually pastebin is my first choice as well and haven’t used others for quite some time now.

                      Heard about

                      https://www.zippyshare.com/
                      https://www.sendspace.com/

                      should be good and anonymous but haven’t tried it so far.

                      Cheers
                      Claudia

                      1 Reply Last reply Reply Quote 0
                      • Scott SumnerS
                        Scott Sumner
                        last edited by

                        Yea, I used to be a fan of regular expression replacement when doing this, but with “larger” datasets there always seems like there is so much tweaking and experimentation needed to get it right (for a particular dataset) that it is hardly worth it, unless you like playing with regular expressions all day instead of solving a particular problem and moving on quickly.

                        A Pythonscript solution such as @Claudia-Frank 's seems fine…

                        1 Reply Last reply Reply Quote 1
                        • patrickdrdP
                          patrickdrd
                          last edited by

                          ok, but why results are inconsistent (with large datasets)?

                          @Claudia-Frank, unfortunately I’m not able to access zippyshare either,
                          so I’ll upload to pastebin from home if we don’t find another solution

                          Claudia FrankC Scott SumnerS 2 Replies Last reply Reply Quote 1
                          • Claudia FrankC
                            Claudia Frank @patrickdrd
                            last edited by

                            @patrickdrd

                            I installed the ue trial version but can’t find the menu item to delete the duplicates.
                            Is there anything I need to install in addition or am I blind and don’t see the obvious?

                            Cheers
                            Claudia

                            1 Reply Last reply Reply Quote 0
                            • Claudia FrankC
                              Claudia Frank
                              last edited by

                              ok - found it - obviously blind :-D

                              Cheers
                              Claudia

                              1 Reply Last reply Reply Quote 0
                              • Scott SumnerS
                                Scott Sumner @patrickdrd
                                last edited by

                                @patrickdrd said:

                                ok, but why results are inconsistent (with large datasets)?

                                Various reasons, sometimes a regular expression approach to this needs to be refined to match the data better before it works well. If you search up some other threads on this topic you can trace through the evolution of a regex approach on certain datasets. However, I suspect you just want to get a workable solution and move on…and I fully endorse that. I’m tired of trying to use regex for this kind of thing. :)

                                BTW, @Claudia-Frank going the extra mile…installing UE trial version just to track this down…nice!

                                Claudia FrankC 1 Reply Last reply Reply Quote 0
                                • Claudia FrankC
                                  Claudia Frank @Scott Sumner
                                  last edited by

                                  @Scott-Sumner

                                  … boots are made for walking … :-)

                                  I’m confused about how UE does sort and delete duplicates.
                                  Used default settings.

                                  Only sorting

                                  Sorting and deleting duplicates

                                  There is obviously something wrong about UEs algorithm, isn’t it?
                                  And the version I used cut the list to 63732 lines.

                                  Cheers
                                  Claudia

                                  1 Reply Last reply Reply Quote 0
                                  • patrickdrdP
                                    patrickdrd
                                    last edited by

                                    so… another number? amazing!

                                    I’ve got version 16 which is lite somehow

                                    1 Reply Last reply Reply Quote 0
                                    • Claudia FrankC
                                      Claudia Frank
                                      last edited by

                                      me too - that is actually the latest one available for ubuntu.

                                      Cheers
                                      Claudia

                                      1 Reply Last reply Reply Quote 0
                                      • patrickdrdP
                                        patrickdrd
                                        last edited by

                                        so the issue is: which result is the correct?

                                        1 Reply Last reply Reply Quote 0
                                        • guy038G
                                          guy038
                                          last edited by guy038

                                          Hello, @patrickdrd, and All,

                                          First of all, I quite pleased to be back on our N++ forum ! Indeed, I was away because of a general failure of my laptop hard drive C:, which, you could imagine, highly annoyed me and needed immediate care :-(((

                                          So, after more than a week, a software purchase of EaseUS Recovery Wizard to restore my data ( the only one which could identify all my files, although Windows could not see the C: partition ), the fact that my first 32 Gb USB key did not work, either, ( Windows cannot format it !!), the re-install of the system on an other hard-disk, the Services Pack, the .NET versions, the different updates, some software installations and the total re-organization of my data structure, I can, now, close that bloody sub-routine !! ( Just note that, if I haven’t been able to get all my files back, I still had a last general backup, performed on the 04/13/18 )


                                          Now, @patrickdrd, you said, in a post above :

                                          guy038 regular expression results in 28109 lines

                                          So, I download the list, from your link, in a new N++ tab :

                                          https://easylist.to/easylist/easylist.txt

                                          Notice that I, personally, found 69889 lines

                                          Now, without any change on that text, I simply performed a sort with the N++ command Edit > Line Operations > Sort Lines Lexicographically Ascending, on that raw text

                                          Then, I used the regex S/R, that I spoke in my previous post :

                                          SEARCH : (?-s)(^.+\R)\1+

                                          REPLACE : \1

                                          I obtained a file of 69790 lines => The difference of 99 lines ( the duplicate ones ) were suppressed, almost immediately !

                                          So, may be, I’m missing something ?

                                          Cheers,

                                          guy038

                                          1 Reply Last reply Reply Quote 1
                                          • patrickdrdP
                                            patrickdrd
                                            last edited by

                                            I can’t say for sure, is there any tool that can tell us how many the unique lines are?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            The Community of users of the Notepad++ text editor.
                                            Powered by NodeBB | Contributors