Community
    • Login

    sort file removing duplicates possible?

    Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
    75 Posts 5 Posters 45.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Claudia FrankC
      Claudia Frank @Scott Sumner
      last edited by

      @Scott-Sumner

      … boots are made for walking … :-)

      I’m confused about how UE does sort and delete duplicates.
      Used default settings.

      Only sorting

      Sorting and deleting duplicates

      There is obviously something wrong about UEs algorithm, isn’t it?
      And the version I used cut the list to 63732 lines.

      Cheers
      Claudia

      1 Reply Last reply Reply Quote 0
      • patrickdrdP
        patrickdrd
        last edited by

        so… another number? amazing!

        I’ve got version 16 which is lite somehow

        1 Reply Last reply Reply Quote 0
        • Claudia FrankC
          Claudia Frank
          last edited by

          me too - that is actually the latest one available for ubuntu.

          Cheers
          Claudia

          1 Reply Last reply Reply Quote 0
          • patrickdrdP
            patrickdrd
            last edited by

            so the issue is: which result is the correct?

            1 Reply Last reply Reply Quote 0
            • guy038G
              guy038
              last edited by guy038

              Hello, @patrickdrd, and All,

              First of all, I quite pleased to be back on our N++ forum ! Indeed, I was away because of a general failure of my laptop hard drive C:, which, you could imagine, highly annoyed me and needed immediate care :-(((

              So, after more than a week, a software purchase of EaseUS Recovery Wizard to restore my data ( the only one which could identify all my files, although Windows could not see the C: partition ), the fact that my first 32 Gb USB key did not work, either, ( Windows cannot format it !!), the re-install of the system on an other hard-disk, the Services Pack, the .NET versions, the different updates, some software installations and the total re-organization of my data structure, I can, now, close that bloody sub-routine !! ( Just note that, if I haven’t been able to get all my files back, I still had a last general backup, performed on the 04/13/18 )


              Now, @patrickdrd, you said, in a post above :

              guy038 regular expression results in 28109 lines

              So, I download the list, from your link, in a new N++ tab :

              https://easylist.to/easylist/easylist.txt

              Notice that I, personally, found 69889 lines

              Now, without any change on that text, I simply performed a sort with the N++ command Edit > Line Operations > Sort Lines Lexicographically Ascending, on that raw text

              Then, I used the regex S/R, that I spoke in my previous post :

              SEARCH : (?-s)(^.+\R)\1+

              REPLACE : \1

              I obtained a file of 69790 lines => The difference of 99 lines ( the duplicate ones ) were suppressed, almost immediately !

              So, may be, I’m missing something ?

              Cheers,

              guy038

              1 Reply Last reply Reply Quote 1
              • patrickdrdP
                patrickdrd
                last edited by

                I can’t say for sure, is there any tool that can tell us how many the unique lines are?

                1 Reply Last reply Reply Quote 0
                • Claudia FrankC
                  Claudia Frank
                  last edited by Claudia Frank

                  Ok - the difference of the total amount of lines in TextFX /python and regex
                  can be explained, the different usage of caseing(not sure if this is the right word).

                  Regex search/replace is insensitive whereas python script is sensitive.
                  Once we use the same, the result is the same.

                  For the provided example list, I assume, insensitive search and deletion is ok
                  but for other cases, sensitive search/replaces might be important.

                  Btw. I also used sort and uniq command line tools from linux with same results as python and regex.

                  Cheers
                  Claudia

                  1 Reply Last reply Reply Quote 2
                  • patrickdrdP
                    patrickdrd
                    last edited by

                    thanks for clarifying, in terms of speed?

                    can python script be made to ignore case?

                    I want to ignore case and I want it fast (useful for large files) too

                    Scott SumnerS 1 Reply Last reply Reply Quote 0
                    • patrickdrdP
                      patrickdrd
                      last edited by patrickdrd

                      these are my settings:
                      https://i.imgur.com/AYmj5I4.jpg

                      and my version:
                      https://i.imgur.com/YRdLkSl.png

                      Claudia FrankC 1 Reply Last reply Reply Quote 0
                      • Scott SumnerS
                        Scott Sumner @patrickdrd
                        last edited by

                        @patrickdrd

                        can python script be made to ignore case?

                        yes

                        I want to ignore case

                        Understood! :-D

                        and I want it fast (useful for large files) too

                        Ok, so what is “fast”? PS is in general going to be slower than some other methods, but unless you sort “large files”–also not well defined-- constantly why is speed all that important?

                        As an example, I took @Claudia-Frank 's PS one-liner above and ran it on the “easylist” data file, and for me it took 3.94 seconds…would that be defined as “fast” or “slow” or…?

                        And on the topic of “large” files, when you get too large (which honestly isn’t really big) you get into trouble with Notepad++ itself dealing with the files…

                        patrickdrdP 1 Reply Last reply Reply Quote 2
                        • Claudia FrankC
                          Claudia Frank @patrickdrd
                          last edited by

                          @patrickdrd

                          yes, with python we could do almost everything :-D

                          Is it important to keep the original lines untouched?
                          I mean, having two lines like

                          Test_line_content
                          test_Line_content
                          

                          could result in either

                          test_line_content
                          

                          or

                          Test_line_content
                          

                          I will post a performance optimized script later today.

                          Cheers
                          Claudia

                          1 Reply Last reply Reply Quote 2
                          • patrickdrdP
                            patrickdrd @Scott Sumner
                            last edited by

                            @Scott-Sumner said:

                            @patrickdrd

                            can python script be made to ignore case?

                            yes

                            I want to ignore case

                            Understood! :-D

                            and I want it fast (useful for large files) too

                            Ok, so what is “fast”? PS is in general going to be slower than some other methods, but unless you sort “large files”–also not well defined-- constantly why is speed all that important?

                            As an example, I took @Claudia-Frank 's PS one-liner above and ran it on the “easylist” data file, and for me it took 3.94 seconds…would that be defined as “fast” or “slow” or…?

                            And on the topic of “large” files, when you get too large (which honestly isn’t really big) you get into trouble with Notepad++ itself dealing with the files…

                            3.94 for a 60k lines file is ok,
                            I mean I show much more time needed, if I remember correctly it should be with regular expressions

                            sorting is ok too, I want them sorted each time, if that’s what you ask scott

                            @claudia, if possible, make them “accent insensitive” too

                            1 Reply Last reply Reply Quote 0
                            • Claudia FrankC
                              Claudia Frank
                              last edited by

                              @Patrick

                              sorry, don’t know the term “accent insensitive” , what does it mean?
                              For example that è is the same as e?

                              Can you provide example data (just need a couple of lines) to see if it is working correctly?
                              The speed test I will do with the easylist text.

                              Cheers
                              Claudia

                              patrickdrdP 1 Reply Last reply Reply Quote 0
                              • Scott SumnerS
                                Scott Sumner
                                last edited by

                                I can envision the following specification for a general purpose script (goes beyond what @patrickdrd has asked for):

                                • sort lines case sensitive, keep duplicate lines
                                • sort lines case sensitive, remove duplicate lines
                                • sort lines case sensitive, keep duplicate lines, reverse order on the sort
                                • sort lines case sensitive, remove duplicate lines, reverse order on the sort
                                • sort lines case insensitive, keep duplicate lines
                                • sort lines case insensitive, remove duplicate lines
                                • sort lines case insensitive, keep duplicate lines, reverse order on the sort
                                • sort lines case insensitive, remove duplicate lines, reverse order on the sort
                                • no sort, remove duplicate lines (case sensitive)
                                • no sort, remove duplicate lines (case insensitive)

                                A selection active when invoking should define:

                                • lines to be affected (only those touched by selection vertically); act on all lines if no selection
                                • columns to be used as the sort key (if rectangular selection use the selected columns as the sort key; if zero-width rect selection consider the key to start at the selection column out to the end of the variable-length lines)

                                Probably I’ve forgotten something important to this “sort” of thing…

                                :-D

                                Scott SumnerS 1 Reply Last reply Reply Quote 1
                                • patrickdrdP
                                  patrickdrd @Claudia Frank
                                  last edited by

                                  @Claudia-Frank said:

                                  @Patrick

                                  sorry, don’t know the term “accent insensitive” , what does it mean?
                                  For example that è is the same as e?

                                  Can you provide example data (just need a couple of lines) to see if it is working correctly?
                                  The speed test I will do with the easylist text.

                                  Cheers
                                  Claudia

                                  yes, exactly that

                                  1 Reply Last reply Reply Quote 1
                                  • Claudia FrankC
                                    Claudia Frank
                                    last edited by

                                    OK - let’s see what we can do.

                                    Cheers
                                    Claudia

                                    1 Reply Last reply Reply Quote 1
                                    • Scott SumnerS
                                      Scott Sumner @Scott Sumner
                                      last edited by

                                      @Scott-Sumner

                                      …the following spec…

                                      Hey Scott! You did forget some things! How about when removing duplicates, we need the options to:

                                      • keep one occurrence of a duplicated line (when sorting)
                                      • keep no occurrences of a duplicated line (when sorting or not sorting)
                                      • keep LAST occurrence of a duplicated line (when not sorting)
                                      • keep FIRST occurrence of a duplicated line (when not sorting)
                                      Claudia FrankC 1 Reply Last reply Reply Quote 0
                                      • Claudia FrankC
                                        Claudia Frank @Scott Sumner
                                        last edited by

                                        @Scott-Sumner

                                        ahh - sorry too late - specs already defined for version 1 you need to open a feature request for version 2 :-D

                                        Cheers
                                        Claudia

                                        1 Reply Last reply Reply Quote 4
                                        • Claudia FrankC
                                          Claudia Frank
                                          last edited by Claudia Frank

                                          You can find the 1st version of the script here.

                                          In order to make it run there are two requirements, apart from the obvious one that you need to have python script plugin installed, which needs to be full-filled.

                                          1.) be sue you have either installed the full package or download and unzip the TclTk into the NPP_INSTALL_DIR. Latest releases

                                          2.) in order to make the “accent insensitive” feature working it is needed to install a python library called unidecode.
                                          Unzip the .whl package into NPP_INSTALL_DIR\plugins\lib\

                                          To check both requirements, open the python script console and do the following commands

                                           import Tkinter
                                           import unidecode
                                          

                                          If you don’t see any errors - done.
                                          Usage is simple - run the script and check the different options.

                                          What should work is

                                          • sort/delete duplicates on whole text (aka nothing is selected)
                                          • sort/delete duplicates on vertically selected text

                                          not supported yet:

                                          • sort/delete duplicates on rectangular selection

                                          Cheers
                                          Claudia

                                          Btw. I spent most of the time creating this ugly window - so if someone wants to create a nicer gui - please go for it. I’m not really good in designing UIs.

                                          1 Reply Last reply Reply Quote 2
                                          • patrickdrdP
                                            patrickdrd
                                            last edited by

                                            tkinter doesn’t work:

                                            Python 2.7.6-notepad++ r2 (default, Apr 21 2014, 19:26:54) [MSC v.1600 32 bit (Intel)]
                                            Initialisation took 219ms
                                            Ready.

                                            import Tkinter
                                            Traceback (most recent call last):
                                            File “<console>”, line 1, in <module>
                                            ImportError: No module named Tkinter
                                            Traceback (most recent call last):
                                            File “D:\Utilities\PortableApps\Notepad++\plugins\PythonScript\scripts\Sorter.py”, line 5, in <module>
                                            import Tkinter as tk
                                            ImportError: No module named Tkinter

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            The Community of users of the Notepad++ text editor.
                                            Powered by NodeBB | Contributors