Is there a way to search for duplicate records in Notepad++?

Adam Buxbaum

Hi All,

I currently and using Notepad++ to review user files before uploading into our provisioning system and I was curious if there was a way to search for duplicates (emails, UID’s, etc…) within Notepad++ or do I have to save the file and review it in excel to do this?

All assistance is greatly appreciated.

Best,
Adam

tomas-chrastina

Hi Adam,

I don’t think there is a way in meaning of search. If you don’t think like use Smart Highlighting, CTRL+F3 or just search.

But there’s a way to remove duplicates from simple list without excel (I use it a lot). So if you have simple list of values:

value1
value2
value2
value3
value2

you can simply get list of unique values:

value1
value2
value3

like this:

You need plugin TextFX Characters
Backup your current editing file !!!
Set TextFX: Menu -> TextFX -> TextFX Tools:
✓ +Sort ascending
✓ +Sort outputs only UNIQUE (at column) lines
Select text
Use one of the actions: Menu -> TextFX -> TextFX Tools:
a) Sort lines case sensitive (at column)
b) Sort lines case insensitive (at column)
Remember to DISABLE option +Sort outputs only UNIQUE (at column) lines, so you won’t lose data when just sorting later!

Still it won’t work for some complex multi-column data, where only Excel filters/remove duplicates of specific data will help.

Best regards,
Tomas

rajeshp2408

Thanks really helped…:-)

Matthias Heim

[Adding my own answer, since this answer gets so many views and was the top result on google]
There is no need to use a plugin.

You can easily find duplicate lines with the following regex:
^([^\r\n]+)$(?=.*?^\1$)

This will find the all occurrences of duplicate lines except the last, so you can also use search and replace to delete them.

You can see it in action here: https://regex101.com/r/5GPJfz/1

Just make sure that you activate the option “. finds \r and \n” in the search-dialogue.

Alan Kilborn

@Matthias-Heim

For me, I like this one to do the same thing:

^((?-s).+?)\R(?=(?s).*?^\1(?:\R|\z))

It has (at least) two advantages:

You don’t have to care about the state of the . matches newline box
The last line of the file doesn’t have to have a line-ending on it to be considered in the duplicate decision (the text itself decides that) – whether it is truly a duplicate then is up for debate, but I think it is

guy038

Hello, @matthias-heim, @alan-kilborn and All,

Alan, I don’t think that the lazy quantifier, at beginning of the regex is necessary, as, obviously, the EOL chars must be matched, anyway !

Hence, the syntax :

(?-s)^(.+)\R(?=(?s).*?^\1(?:\R|\z))

However, @matthias-heim be aware that in case of an important amount of lines between current line scanned and its nearest duplicate, the regex may completely fail to detect correct matches :-((

Best Regards

guy038

Mohammed Asif

@guy038 said in Is there a way to search for duplicate records in Notepad++?:

(?-s)^(.+)\R(?=(?s).*?^\1(?:\R|\z))

can you please tell me how to mark both lines (original+duplicate)?

guy038

Hello, @mohammed-asif and All,

Before practically answering to your question, could you tell us some hints about your data :

Why do you want to mark all the duplicate lines ? Do you intend to delete them all or copy them for other process or else ?
How many lines, about, to processed and the average length of the lines ?
How many lines, max, about, between two duplicate lines ?

May be, you could add a short example of your text ?

I’ve already found out a solution but it mainly depends on the data’s organization and on what kind of process is needed after bookmarking !

See you later,

Best Regards,

guy038

Alan Kilborn

@guy038 said in Is there a way to search for duplicate records in Notepad++?:

Why do you want to mark all the duplicate lines ?

A practical reason (not involving copy/cut/delete) for this might be so that each duplicate line can be visited and manually edited to be made unique in some way.