@Matthew-Owen @PeterJones
It’s actually perfectly feasible to remove all non-empty lines that are duplicated anywhere in the file, regardless of order, without using plugins, although this unfortunately can’t be saved in a macro. This is an extension of the “group lines by content and then put them back in order” procedure that I’ve also used to empty all duplicate lines (leaving them empty) efficiently. The general “group lines by content and then put them back in order” procedure is just steps 1-5 and 7-10 below, where step 6 is the part where you do whatever you want with the lines while they’re grouped by content.
For all of the below find/replace operations, Wrap around must be checked, and Search Mode must be set to Regular expression.
The steps are as follows:
Replace ^ with \x20 (add a space at the beginning of each line) Move the caret to the start of the file Use the column editor to add line numbers at the start of each line: Number to Insert checked Initial number, Increase by, Repeat all set to 1 Leading = Zeros Format: Dec Replace (?-s)^(\d+)\x20(.*) with ${2}\x20${1} (swap the line numbers with the rest of the line, so that unique lines are annotated with their location) Run the command Edit->Line Operations->Sort Lines Lexicographically Ascending Replace ^(.+?)\x20\d+(\R)(?:\1\x20\d+\R?)+ with nothing (remove all non-empty lines that aren’t unique, replace the .+? with .*? if you also want to remove empty lines) Replace ^(.*?)\x20(\d+)$ with ${2}\x20${1} (move the line numbers back to the start of the line) Remove the empty line from the end of the file, if there is one Run the command Edit->Line Operations->Sort Lines Lexicographically Ascending again (put lines back in the original order) Replace ^\d+\x20 with nothing (remove the starting line numbers and space)As a demonstration, this procedure can take the file
1337 1 bar 1337 1 baz jke quz 1337 1 quz nhj blah baz jke 103and convert it into
bar nhj blah 103