@guy038 said in About single and duplicate lines...:
About single and duplicate lines…
General Discussion
1 posts
1 posters
225 views
guy038
Mar 24, 2026, 8:45 PM
Hello, All,
Reading that post made me realize that searching for single or duplicate lines is a very common task. Some time ago, for my personal workflow, I had written a method to solve the main cases ! So, in this post, I’m going to show you, from an original file, how to keep :
All single lines, ONLY
All duplicate lines, ONLY
All single lines and the first copy of all duplicate lines
All single lines and the last copy of all duplicate lines
The first copy of all duplicate lines, ONLY
The last copy of all duplicate lines, ONLY
I’ll use a file, named Test_File.txt, that both contains single lines and duplicate lines that appear in 2, 3, 4 or more times. It contains 48 color palettes, found from various sites and added one after another, giving a total of 78,117 records whose 39,532 are single lines and 38,585 are duplicate lines. On the other hand, if we count one copy of all the duplicates, this file contains 11,290 different duplicate lines.
To test my solutions, simply download this UTF-8 file ( 5,937,560 bytes ) from my Google Drive account :
https://drive.google.com/file/d/1aYOpKon4KYw_NXSdj4Tm4Ti_FrygC2ky/view?usp=sharing
Remarks :
Note the definition of single lines : these are lines that differ in characters and/or case from all the other lines of the current file. For example, in this small file of 14 lines, below :
ABC
xyz
123
789
HIJ
HIJ
123
AbC
123
HIJ
abc
HIJ
456
xyz
The 5 lines ABC, AbC, abc, 789 and 456 are considered to be single lines, as different in chars and/or case from all the other lines.
The 3 123 lines are considered to be a duplicate line with 3 copies ( Multiple occurrences )
The 2 xyz lines are considered to be a duplicate line with 2 copies ( Multiple occurrences )
Les 4 HIJ lines are considered to be a duplicate line with 4 copies ( Multiple occurrences )
IMPORTANT :
I’ve done some of the work for you, by adding a final column that numbers all lines in this file. Thus, is will be easy to restore the original order of the remaining records, after that each processing is complete. So, in case you need this initial order :
Put the caret right before the present number, at the end of the first line
Run the Edit > Begin/End Select in Column Mode option ( or use the Alt + Shift + B shortcut )
Move to the last line of the file
Put the caret right before the present number, at the end of the last line
Run again the Edit > Begin/End Select in Column Mode option ( or use the Alt + Shift + B shortcut )
=> A ZERO-LINE column mode selection should appear throughout all the lines
Then, run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
=> The original order of the remaining records, AFTER completion of one of the 6 methods below, should be back !
In each procedure, below, 1 or 2 S/R are used. To process them :
First, cancel any existing selection to ensure that any line-end character will be taken in account during the S/R phase
Open the Replace dialog ( Ctrl + H )
Uncheck all box options
Check the Wrap around option
Select the Regular expression search mode
Click on the Replace All button
(1) To keep all the SINGLE lines ONLY ( 39,532 records ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+
REPLACE Leave EMPTY
Perform the IMPORTANT section, above
(2) To keep all the DUPLICATE lines ONLY ( 38,585 records = 78,117 - 39,532 ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R
REPLACE Leave EMPTY
Perform the IMPORTANT section, above
(3) To keep all the SINGLE lines and the FIRST copy of ALL the DUPLICATE lines, found AFTER the sort ( 50,822 records ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( ( .+ ) .{7} \R ) (?: \2 .{7} \R )+
REPLACE \1
Perform the IMPORTANT section, above
(4) To keep all the SINGLE lines and the LAST copy of all the DUPLICATE lines, found AFTER the sort ( 50,822 records ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )* ( \1 .{7} \R )
REPLACE \2
Perform the IMPORTANT section, above
(5) To keep the FIRST copy of all the DUPLICATE lines ONLY, found AFTER the sort ( 11,290 = 50,822 - 39,532 ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R
REPLACE Leave EMPTY
Then :
SEARCH (?x-is) ^ ( ( .+ ) .{7} \R ) (?: \2 .{7} \R )+
REPLACE \1
Perform the IMPORTANT section, above
(6) To keep the LAST copy of all the DUPLICATE lines ONLY, found AFTER the sort ( 11,290 = 50,822 - 39,532 ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab and select all text ( Ctrl + A )
Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option
Click anywhere, in the new tab, to cancel the entire selection
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R
REPLACE Leave EMPTY
Then :
SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )* ( \1 .{7} \R )
REPLACE \2
Perform the IMPORTANT section, above
At the very end of any of these choices, you may delete the extra numeration :
SEARCH (?x-s) .{7} $
REPLACE Leave EMPTY
Then run the Edit > Blank Operations > Trim Trailing Space
Best Regards,
guy038
P.S. :
Note that there is also a native way to get all the single lines and the first copy of all the duplicate lines, found with the present order ( 50,822 records ) :
Paste the Text_File.txt contents in a new tab
Switch to that new tab
Delete the numeration, at end of each line :
SEARCH (?x-s) .{7} $
REPLACE Leave EMPTY
Then, use the Edit > Line Opérations > Remove Duplicate lines option
That’s a pretty solid breakdown 👍
For most cases though, I’d honestly just go with the built-in “Remove Duplicate Lines” unless you specifically need first/last occurrences. Way simpler and less error-prone.
The regex approach is powerful, but yeah… a bit overkill unless you’re dealing with very specific cases or large datasets.
