@guy038 proposed a very simple solution to this, but I have a multi-step solution using no plugins, without having to perform the same find-replace multiple times in a row:
Make sure Regular expressions are ON in the find/replace form before starting this.
The document starts looking like this:
foo
bar
baz
foo
foo
bar
bar
baz
quz
foo
Use the find/replace form to replace ^ with \x07.
This adds a BEL character (convenient because it does not show up naturally in most text documents) to the beginning of each line.
Make a column selection in the first column of every line of the file, then use the column editor to insert a number (with leading zeros to facilitate sorting).
This numbers the rows, so they can later be put back in order.
6e8abd36-ff51-41a8-96d7-7306ab623669-image.png
Use the find/replace form to replace (?-s)^(\d+)(\x07)(.*) with ${3}${2}${1}.
This puts the column numbers after everything else.
Use the menu command Edit->Line Operations->Sort Lines Lex. Ascending Ignoring Case.
Now all the lines with the same text are grouped together, and a single regex-replace can get rid of all but the first line with given text.
3c2274bc-a772-4b8f-8f5d-2f4f7149f143-image.png
Find/replace (?-s)^(.+)\x07\d+(?:\R\1\x07\d+)* with ~~~~${0}.
This marks the first instance of each line, so that it won’t be deleted later. Note that the ~~~~ in this example should be replaced with some other text that occurs nowhere in your document.
c755e9c0-d0f2-4c5d-bd1c-cb8a5b008180-image.png
Find/replace (?-s)^(?!~~~~).*(\x07\d+) with ${1}.
This clears the text but not the line number of any line that does not have the first instance of its text.
1d2fdb73-56c6-48f7-a59b-0a41ee2181a1-image.png
Find/replace ^~~~~ with nothing.
This removes the starting marker.
Find/replace (?-s)^(.*)(\x07)(\d+)$ with ${3}${2}$1.
This brings the line numbers back to the front so that the lines can go back in order.
Use the menu command Edit->Line Operations->Sort Lines Lex. Ascending Ignoring Case.
Now all the lines are back in their original order.
15e0de09-df0e-4c9a-97b7-0e0314d296dc-image.png
Find/replace ^\d+\x07 with nothing.
This removes the line numbers and separator.
Finally, you are left with the original document with the non-first instances of each line’s text replaced with nothing! A lot of steps, but every step is highly scalable and won’t exhibit bad performance on very large files.
foo
bar
baz
quz