sort file removing duplicates possible?
-
so the issue is: which result is the correct?
-
Hello, @patrickdrd, and All,
First of all, I quite pleased to be back on our N++ forum ! Indeed, I was away because of a general failure of my laptop hard drive
C:, which, you could imagine, highly annoyed me and needed immediate care :-(((So, after more than a week, a software purchase of EaseUS Recovery Wizard to restore my data ( the only one which could identify all my files, although Windows could not see the
C:partition ), the fact that my first 32 Gb USB key did not work, either, ( Windows cannot format it !!), the re-install of the system on an other hard-disk, the Services Pack, the .NET versions, the different updates, some software installations and the total re-organization of my data structure, I can, now, close that bloody sub-routine !! ( Just note that, if I haven’t been able to get all my files back, I still had a last general backup, performed on the 04/13/18 )
Now, @patrickdrd, you said, in a post above :
guy038 regular expression results in 28109 lines
So, I download the list, from your link, in a new N++ tab :
https://easylist.to/easylist/easylist.txt
Notice that I, personally, found
69889linesNow, without any change on that text, I simply performed a sort with the N++ command
Edit > Line Operations > Sort Lines Lexicographically Ascending, on that raw textThen, I used the regex S/R, that I spoke in my previous post :
SEARCH :
(?-s)(^.+\R)\1+REPLACE :
\1I obtained a file of
69790lines => The difference of99lines ( theduplicateones ) were suppressed, almost immediately !So, may be, I’m missing something ?
Cheers,
guy038
-
I can’t say for sure, is there any tool that can tell us how many the unique lines are?
-
Ok - the difference of the total amount of lines in TextFX /python and regex
can be explained, the different usage of caseing(not sure if this is the right word).Regex search/replace is insensitive whereas python script is sensitive.
Once we use the same, the result is the same.For the provided example list, I assume, insensitive search and deletion is ok
but for other cases, sensitive search/replaces might be important.Btw. I also used sort and uniq command line tools from linux with same results as python and regex.
Cheers
Claudia -
thanks for clarifying, in terms of speed?
can python script be made to ignore case?
I want to ignore case and I want it fast (useful for large files) too
-
these are my settings:
https://i.imgur.com/AYmj5I4.jpgand my version:
https://i.imgur.com/YRdLkSl.png -
can python script be made to ignore case?
yes
I want to ignore case
Understood! :-D
and I want it fast (useful for large files) too
Ok, so what is “fast”? PS is in general going to be slower than some other methods, but unless you sort “large files”–also not well defined-- constantly why is speed all that important?
As an example, I took @Claudia-Frank 's PS one-liner above and ran it on the “easylist” data file, and for me it took 3.94 seconds…would that be defined as “fast” or “slow” or…?
And on the topic of “large” files, when you get too large (which honestly isn’t really big) you get into trouble with Notepad++ itself dealing with the files…
-
yes, with python we could do almost everything :-D
Is it important to keep the original lines untouched?
I mean, having two lines likeTest_line_content test_Line_contentcould result in either
test_line_contentor
Test_line_contentI will post a performance optimized script later today.
Cheers
Claudia -
@Scott-Sumner said:
can python script be made to ignore case?
yes
I want to ignore case
Understood! :-D
and I want it fast (useful for large files) too
Ok, so what is “fast”? PS is in general going to be slower than some other methods, but unless you sort “large files”–also not well defined-- constantly why is speed all that important?
As an example, I took @Claudia-Frank 's PS one-liner above and ran it on the “easylist” data file, and for me it took 3.94 seconds…would that be defined as “fast” or “slow” or…?
And on the topic of “large” files, when you get too large (which honestly isn’t really big) you get into trouble with Notepad++ itself dealing with the files…
3.94 for a 60k lines file is ok,
I mean I show much more time needed, if I remember correctly it should be with regular expressionssorting is ok too, I want them sorted each time, if that’s what you ask scott
@claudia, if possible, make them “accent insensitive” too
-
@Patrick
sorry, don’t know the term “accent insensitive” , what does it mean?
For example that è is the same as e?Can you provide example data (just need a couple of lines) to see if it is working correctly?
The speed test I will do with the easylist text.Cheers
Claudia -
I can envision the following specification for a general purpose script (goes beyond what @patrickdrd has asked for):
- sort lines case sensitive, keep duplicate lines
- sort lines case sensitive, remove duplicate lines
- sort lines case sensitive, keep duplicate lines, reverse order on the sort
- sort lines case sensitive, remove duplicate lines, reverse order on the sort
- sort lines case insensitive, keep duplicate lines
- sort lines case insensitive, remove duplicate lines
- sort lines case insensitive, keep duplicate lines, reverse order on the sort
- sort lines case insensitive, remove duplicate lines, reverse order on the sort
- no sort, remove duplicate lines (case sensitive)
- no sort, remove duplicate lines (case insensitive)
A selection active when invoking should define:
- lines to be affected (only those touched by selection vertically); act on all lines if no selection
- columns to be used as the sort key (if rectangular selection use the selected columns as the sort key; if zero-width rect selection consider the key to start at the selection column out to the end of the variable-length lines)
Probably I’ve forgotten something important to this “sort” of thing…
:-D
-
@Claudia-Frank said:
@Patrick
sorry, don’t know the term “accent insensitive” , what does it mean?
For example that è is the same as e?Can you provide example data (just need a couple of lines) to see if it is working correctly?
The speed test I will do with the easylist text.Cheers
Claudiayes, exactly that
-
OK - let’s see what we can do.
Cheers
Claudia -
…the following spec…
Hey Scott! You did forget some things! How about when removing duplicates, we need the options to:
- keep one occurrence of a duplicated line (when sorting)
- keep no occurrences of a duplicated line (when sorting or not sorting)
- keep LAST occurrence of a duplicated line (when not sorting)
- keep FIRST occurrence of a duplicated line (when not sorting)
-
ahh - sorry too late - specs already defined for version 1 you need to open a feature request for version 2 :-D
Cheers
Claudia -
You can find the 1st version of the script here .
In order to make it run there are two requirements, apart from the obvious one that you need to have python script plugin installed, which needs to be full-filled.
1.) be sue you have either installed the full package or download and unzip the TclTk into the NPP_INSTALL_DIR. Latest releases
2.) in order to make the “accent insensitive” feature working it is needed to install a python library called unidecode .
Unzip the .whl package into NPP_INSTALL_DIR\plugins\lib\To check both requirements, open the python script console and do the following commands
import Tkinter import unidecodeIf you don’t see any errors - done.
Usage is simple - run the script and check the different options.What should work is
- sort/delete duplicates on whole text (aka nothing is selected)
- sort/delete duplicates on vertically selected text
not supported yet:
- sort/delete duplicates on rectangular selection
Cheers
ClaudiaBtw. I spent most of the time creating this ugly window - so if someone wants to create a nicer gui - please go for it. I’m not really good in designing UIs.
-
tkinter doesn’t work:
Python 2.7.6-notepad++ r2 (default, Apr 21 2014, 19:26:54) [MSC v.1600 32 bit (Intel)]
Initialisation took 219ms
Ready.import Tkinter
Traceback (most recent call last):
File “<console>”, line 1, in <module>
ImportError: No module named Tkinter
Traceback (most recent call last):
File “D:\Utilities\PortableApps\Notepad++\plugins\PythonScript\scripts\Sorter.py ”, line 5, in <module>
import Tkinter as tk
ImportError: No module named Tkinter -
I found out something else, about that easylist file,
textfx’s case insensitive sort results in 69234 entries,
which is the same as ue’s result! -
Hello, @patrickdrd, and All,
Well, I must complete my previous post !
- Firstly, I realized that your list, below, is constantly updated ( Last modified: 02 Jun 2018 08:09 UTC )
https://easylist.to/easylist/easylist.txt
So, today, this list contains
69917lines
- Secondly, when performing the regex S/R, we must consider, both, sensitive and insensitive search => The two search regexes :
Regex A :
(?-is)(^.+\R)\1+Regex B :
(?i-s)(^.+\R)\1+give, after sorting and removing duplicates with the regex, a file containing :
A
69852lines ( so,65lines deleted, in56matches )B
69817lines ( so,100lines deleted, in88matches )
- Thirdly, we, also, must take in account the possibility that the sort, itself, is run in a sensitive or insensitive way !
Natively, Notepad++ sort text, according to the Unicode value ( code-point ) of characters ( a kind of sensitive sort ! ) whereas some other text editors may consider these two case options, leading to different results !
For instance, using the RJ TextEd software, here are the differences with a simple list of three-characters strings ( 1 x ‘ABC’, 2 x ‘AbC’, 3 x ‘Abc’,1 x ‘DEF’, 3 x ‘DEf’, 2 x ‘aBC’, 3 x ‘aBc’, 3 x ‘dEF’ and 3 x ‘def’ )
•-----------------------•---------------------------• | with Notepad++ | with RJ TextEd | •---------------•-----------------------•---------------------------• | Before Sort | After UNICODE Sort | After SENSITIVE Sort | •---------------•-----------------------•---------------------------• | Abc | ABC | AbC | | Abc | AbC | aBC | | DEf | AbC | aBc | | AbC | Abc | Abc | | aBc | Abc | ABC | | dEF | Abc | aBC | | aBc | DEF | Abc | | DEf | DEf | AbC | | aBC | DEf | aBc | | def | DEf | aBc | | AbC | aBC | Abc | | def | aBC | DEf | | def | aBc | dEF | | ABC | aBc | dEF | | Abc | aBc | DEf | | DEF | dEF | dEF | | aBc | dEF | DEf | | dEF | dEF | def | | dEF | def | DEF | | DEf | def | def | | aBC | def | def | •---------------•-----------------------•---------------------------•So, it’s easy to understand that removing consecutive duplicates, after the sort, with the regexes above, will, necessarily, give results totally different, depending of the software used :-(
- Fourthly, sort may give different results, after being run several times, one after another. For instance, with RJ TextEd, running
6times the insensitive sort on the3character list above, I was left with3sets of data (4times, identical to the sensitive sort and two other lists !! Luckily, as for Notepad++, its Unicode sort always give identical results :-))
That’s why, @patrickdrd, it’s very difficult, finally, to compare results between different softwares, at each piece have its own behavior !
Cheers,
guy038
P.S. :
Here are the results of my tests :
1) With Notepad++ and RJ TextEd, using sensitive sort :
•---------------------------------------------------------------------------------------------------• | with Notepad++ Features | •---------------•-------------------------•---------------------------•-----------------------------• | Before Sort | After Sensitive Sort | After Sensitive Regex + | After INsensitive Regex + | | | | Suppression Duplicates | Suppression Duplicates | •---------------•-------------------------•---------------------------•-----------------------------• | Abc | ABC | ABC | ABC | | Abc | AbC | AbC | DEF | | DEf | AbC | Abc | aBC | | AbC | Abc | DEF | dEF | | aBc | Abc | DEf | | | dEF | Abc | aBC | | | aBc | DEF | aBc | | | DEf | DEf | dEF | | | aBC | DEf | def | | | def | DEf | | | | AbC | aBC | | | | def | aBC | | | | def | aBc | | | | ABC | aBc | | | | Abc | aBc | | | | DEF | dEF | | | | aBc | dEF | | | | dEF | dEF | | | | dEF | def | | | | DEf | def | | | | aBC | def | | | •---------------•-------------------------•---------------------------•-----------------------------• •---------------------------------------------------------------------------------------------------• | with RJ TextEd Features | •---------------•-------------------------•---------------------------•-----------------------------• | Before Sort | After Sensitive Sort | After Sensitive Regex + | After INsensitive Regex + | | | | Suppression Duplicates | Suppression Duplicates | •---------------•-------------------------•---------------------------•-----------------------------• | Abc | AbC | AbC | AbC | | Abc | aBC | aBC | DEf | | DEf | aBc | aBc | | | AbC | Abc | Abc | | | aBc | ABC | ABC | | | dEF | aBC | aBC | | | aBc | Abc | Abc | | | DEf | AbC | AbC | | | aBC | aBc | aBc | | | def | aBc | Abc | | | AbC | Abc | DEf | | | def | DEf | dEF | | | def | dEF | DEf | | | ABC | dEF | dEF | | | Abc | DEf | DEf | | | DEF | dEF | def | | | aBc | DEf | DEF | | | dEF | def | def | | | dEF | DEF | | | | DEf | def | | | | aBC | def | | | •---------------•-------------------------•---------------------------•-----------------------------•2) When running, several times, an insensitive sort, with RJ TextEd, I obtained
3different lists :-
The first one was identical to the table just above, which uses a sensitive sort
-
The two others are listed below !
•---------------------------------------------------------------------------------------------------• | with RJ TextEd Features | •---------------•-------------------------•---------------------------•-----------------------------• | Before Sort | After INsensitive Sort | After Sensitive Regex + | After INsensitive Regex + | | | | Suppression Duplicates | Suppression Duplicates | •---------------•-------------------------•---------------------------•-----------------------------• | Abc | ABC | ABC | ABC | | Abc | AbC | AbC | dEF | | DEf | aBC | aBC | | | AbC | aBC | aBc | | | aBc | aBc | Abc | | | dEF | Abc | AbC | | | aBc | AbC | aBc | | | DEf | aBc | Abc | | | aBC | Abc | aBc | | | def | Abc | dEF | | | AbC | aBc | DEf | | | def | dEF | def | | | def | dEF | dEF | | | ABC | DEf | DEF | | | Abc | DEf | def | | | DEF | DEf | | | | aBc | def | | | | dEF | def | | | | dEF | dEF | | | | DEf | DEF | | | | aBC | def | | | •---------------•-------------------------•---------------------------•-----------------------------• •---------------------------------------------------------------------------------------------------• | with RJ TextEd Features | •---------------•-------------------------•---------------------------•-----------------------------• | Before Sort | After INsensitive Sort | After Sensitive Regex + | After INsensitive Regex + | | | | Suppression Duplicates | Suppression Duplicates | •---------------•-------------------------•---------------------------•-----------------------------• | Abc | ABC | ABC | ABC | | Abc | AbC | AbC | dEF | | DEf | aBC | aBC | | | AbC | aBC | aBc | | | aBc | aBc | Abc | | | dEF | Abc | aBc | | | aBc | aBc | AbC | | | DEf | AbC | Abc | | | aBC | Abc | aBc | | | def | Abc | dEF | | | AbC | aBc | DEf | | | def | dEF | def | | | def | dEF | DEF | | | ABC | dEF | DEf | | | Abc | DEf | | | | DEF | DEf | | | | aBc | def | | | | dEF | def | | | | dEF | def | | | | DEf | DEF | | | | aBC | DEf | | | •---------------•-------------------------•---------------------------•-----------------------------• -
Patrick, did you downlaod and unzip the TclTk into the
NPP_INSTALL_DIR ? (in your case into D:\Utilities\PortableApps\Notepad++)If so, can you run the following in the python script console
import sys; print '\n'.join(sys.path)and post the output?
Did the unidecode library installation work?
Cheers
Claudia
