Duplicate lines

[Dusty Pulver]

I need to edit a mailing list of over 10,000 users.
Is there a plug-in that will remove duplicate lines?
If not, any ideas of how it can be done easily?

[Scott Sumner]

If you don’t care if the lines get sorted, the quickest way is to select all text in the file and use the TextFX menu, choosing TextFX Tools, enable the “Sort outputs only UNIQUE (at column) lines” option, and then choose a “Sort Lines…” action (from the same menu where that option is from).

[Dusty Pulver]

Remove duplicates, remove blank lines and sort data in one operation

Paste the text into Notepad++ (CTRL+V). ...
Mark all the text (CTRL+A). ...
Check +Sort outputs only UNIQUE (at column) lines (if not already checked) 
Click TextFX → Click TextFX Tools → Click Sort lines case insensitive (at column)
Duplicates and blank lines have been removed and the data has been sorted alphabetically.

[guy038]

Hello, Dusty Pulver,

An other simple possibility, that does not need any plugin, would be :

• Sort your file, with the native N+++ sort ( menu option Edit - Line Operations - Sort Lines Lexicographically Ascending )

• Move to the beginning of your file ( CTRL + Origin )

• Open the Replace dialog ( CTRL + H )

• Perform the S/R, below, in Regular expression search mode :

SEARCH : (?-s)^(.*\R)\1+

REPLACE : \1

Notes :

• The (?-s) modifier ensures that the dot will match standard characters, only, even if you previously checked the . matches newline option !

• The \R stands for any kind of EOL character(s)

Best Regards,

guy038