About single and duplicate lines...

[guy038]

Hello, All,

Reading that post made me realize that searching for single or duplicate lines is a very common task. Some time ago, for my personal workflow, I had written a method to solve the main cases ! So, in this post, I’m going to show you, from an original file, how to keep :

• All single lines, ONLY

• All duplicate lines, ONLY

• All single lines and the first copy of all duplicate lines

• All single lines and the last copy of all duplicate lines

• The first copy of all duplicate lines, ONLY

• The last copy of all duplicate lines, ONLY

I’ll use a file, named Test_File.txt, that both contains single lines and duplicate lines that appear in 2, 3, 4 or more times. It contains 48 color palettes, found from various sites and added one after another, giving a total of 78,117 records whose 39,532 are single lines and 38,585 are duplicate lines. On the other hand, if we count one copy of all the duplicates, this file contains 11,290 different duplicate lines.

To test my solutions, simply download this UTF-8 file ( 5,937,560 bytes ) from my Google Drive account :

https://drive.google.com/file/d/1aYOpKon4KYw_NXSdj4Tm4Ti_FrygC2ky/view?usp=sharing

---

Remarks :

Note the definition of single lines : these are lines that differ in characters and/or case from all the other lines of the current file. For example, in this small file of 14 lines, below :

    ABC
    xyz
    123
    789
    HIJ
    HIJ
    123
    AbC
    123
    HIJ
    abc
    HIJ
    456
    xyz

• The 5 lines ABC, AbC, abc, 789 and 456 are considered to be single lines, as different in *chars and/or case from all the other lines.

• The 3 123 lines are considered to be a duplicate line with 3 copies ( Multiple occurrences )

• The 2 xyz lines are considered to be a duplicate line with 2 copies ( Multiple occurrences )

• Les 4 HIJ lines are considered to be a duplicate line with 4 copies ( Multiple occurrences )

---

IMPORTANT :

I’ve done some of the work for you, by adding a final column that numbers all lines in this file. Thus, is will be easy to restore the original order of the remaining records, after that each processing is complete. So, in case you need this initial order :

• Put the caret right before the number 00001, at the end of the first line

• Run the Edit > Begin/End Select in Column Mode option ( or use the Alt + Shift + B shortcut )

• Move to the last line of the file

• Put the caret right before the number 78117

• Run again the Edit > Begin/End Select in Column Mode option ( or use the Alt + Shift + B shortcut )

=> A ZERO-LINE column mode selection should appear throughout all the lines

• Then, run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

=> The original order of the remaining records should be back !

---

In each procedure, below, 1 or 2 S/R are used. To process them :

• First, cancel any existing selection to ensure that any line-end character will be taken in account during the S/R phase

• Open the Replace dialog ( Ctrl + H )

• Uncheck all box options

• Check the Wrap around option

• Select the Regular expression search mode

• Click on the Replace All button

---

(1) To keep all the SINGLE lines ONLY ( 39,532 records ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+

• REPLACE Leave EMPTY

• Perform the IMPORTANT section, above

---

(2) To keep all the DUPLICATE lines ONLY ( 38,585 records = 78,117 - 39,532 ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R

• REPLACE Leave EMPTY

• Perform the IMPORTANT section, above

---

(3) To keep all the SINGLE lines and the FIRST copy of ALL the DUPLICATE lines, found AFTER the sort ( 50,822 records ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( ( .+ ) .{7} \R ) (?: \2 .{7} \R )+

• REPLACE \1

• Perform the IMPORTANT section, above

---

(4) To keep all the SINGLE lines and the LAST copy of all the DUPLICATE lines, found AFTER the sort ( 50,822 records ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )* ( \1 .{7} \R )

• REPLACE \2

• Perform the IMPORTANT section, above

---

(5) To keep the FIRST copy of all the DUPLICATE lines ONLY, found AFTER the sort ( 11,290 = 50,822 - 39,532 ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R

• REPLACE Leave EMPTY

Then :

• SEARCH (?x-is) ^ ( ( .+ ) .{7} \R ) (?: \2 .{7} \R )+

• REPLACE \1

• Perform the IMPORTANT section, above

---

(6) To keep the LAST copy of all the DUPLICATE lines ONLY, found AFTER the sort ( 11,290 = 50,822 - 39,532 ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab and select all text ( Ctrl + A )

• Run the Edit > Line Operations > Sort Lines Lexicographically Ascending option

• Click anywhere, in the new tab, to cancel the entire selection

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )+ (*SKIP) (*F) | ^ .+ \R

• REPLACE Leave EMPTY

Then :

• SEARCH (?x-is) ^ ( .+ ) .{7} \R (?: \1 .{7} \R )* ( \1 .{7} \R )

• REPLACE \2

• Perform the IMPORTANT section, above

---

At the very end of any of these choices, you may delete the extra numeration :

• SEARCH (?x-s) .{7} $

• REPLACE Leave EMPTY

• Then run the Edit > Blank Operations > Trim Trailing Space

---

Best Regards,

guy038

P.S. :

Note that there is also a native way to get all the single lines and the first copy of all the duplicate lines, found with the present order ( 50,822 records ) :

• Paste the Text_File.txt contents in a new tab

• Switch to that new tab

• Delete the numeration, at end of each line :

  • SEARCH (?x-s) .{7} $

  • REPLACE Leave EMPTY

• Then, use the Edit > Line Opérations > Remove Duplicate lines option