Is there a way to search for duplicate records in Notepad++?
Adam Buxbaum last edited by
I currently and using Notepad++ to review user files before uploading into our provisioning system and I was curious if there was a way to search for duplicates (emails, UID’s, etc…) within Notepad++ or do I have to save the file and review it in excel to do this?
All assistance is greatly appreciated.
tomas-chrastina last edited by tomas-chrastina
I don’t think there is a way in meaning of search. If you don’t think like use Smart Highlighting, CTRL+F3 or just search.
But there’s a way to remove duplicates from simple list without excel (I use it a lot). So if you have simple list of values:
value1 value2 value2 value3 value2
you can simply get list of unique values:
value1 value2 value3
- You need plugin TextFX Characters
- Backup your current editing file !!!
- Set TextFX: Menu -> TextFX -> TextFX Tools:
✓ +Sort ascending
✓ +Sort outputs only UNIQUE (at column) lines
- Select text
- Use one of the actions: Menu -> TextFX -> TextFX Tools:
a) Sort lines case sensitive (at column)
b) Sort lines case insensitive (at column)
- Remember to DISABLE option +Sort outputs only UNIQUE (at column) lines, so you won’t lose data when just sorting later!
Still it won’t work for some complex multi-column data, where only Excel filters/remove duplicates of specific data will help.
rajeshp2408 last edited by
Thanks really helped…:-)
Matthias Heim last edited by
[Adding my own answer, since this answer gets so many views and was the top result on google]
There is no need to use a plugin.
You can easily find duplicate lines with the following regex:
This will find the all occurrences of duplicate lines except the last, so you can also use search and replace to delete them.
You can see it in action here: https://regex101.com/r/5GPJfz/1
Just make sure that you activate the option “. finds \r and \n” in the search-dialogue.
Alan Kilborn last edited by
For me, I like this one to do the same thing:
It has (at least) two advantages:
You don’t have to care about the state of the
. matches newlinebox
The last line of the file doesn’t have to have a line-ending on it to be considered in the duplicate decision (the text itself decides that) – whether it is truly a duplicate then is up for debate, but I think it is
guy038 last edited by
Alan, I don’t think that the lazy quantifier, at beginning of the regex is necessary, as, obviously, the EOL chars must be matched, anyway !
Hence, the syntax :
However, @matthias-heim be aware that in case of an important amount of lines between current line scanned and its nearest duplicate, the regex may completely fail to detect correct matches :-((
Mohammed Asif last edited by
can you please tell me how to mark both lines (original+duplicate)?
guy038 last edited by
Hello, @mohammed-asif and All,
Before practically answering to your question, could you tell us some hints about your data :
Why do you want to mark all the duplicate lines ? Do you intend to delete them all or copy them for other process or else ?
How many lines, about, to processed and the average length of the lines ?
How many lines, max, about, between two duplicate lines ?
May be, you could add a short example of your text ?
I’ve already found out a solution but it mainly depends on the data’s organization and on what kind of process is needed after bookmarking !
See you later,