IMHO, I supposed that your current file contains too much data ! I, very often, verified that complicated regexes totally fail when applied to huge amounts of text with the result that, only, one wrong match of all the file contents, occurs :-((

May be, try to slice your file in smaller parts ! It could help ?!

Generally, this problem often occurs when using recursion feature in regexes. But, it’s quite difficult to fully understand the limitations of the Boost regex engine, used in N++ !

Cheers,

guy038

Find what : (?si)(?:^|\h+)(“?)['.,]?(\w[\w '.-]\w)['.,]?\1(?=(?:\h+|\R)(?:[^{}](?:\h+|\R))?(”?)[‘.,]?\2[’.,]?\3(?:\h|\R))
replacing with nothing

wrap around
regular expression

and its blanking the file… (replace all: 1 occurrence was replaced)

nvm ?..? seems a extra space worked into the code…

OK ! So, I changed the part of the regex , (\w[\w '-]*\w), responsible of matching the name, that is to be deleted. The new regex is ['.,]?(\w[\w '.-]*\w)['.,]?, which means that a name :

• Begins with a word character, possibly preceded by an apostrophe ( ' ), a dot ( . ) or a comma (,) symbols

• Contains, afterwards, a sequence, possibly empty of word characters or an apostrophe ( ' ) , a dot ( . ), an hyphen ( - ) or a space symbol

• And ends with with a word character, possibly followed by an apostrophe ( ' ), a dot ( . ) or a comma ( , ) symbols

Only the inner part, beginning and ending with a word character, is considered as the group 2, which must occur, further on, a second time. Note also, that names, with some leading or trailing symbols, may be surrounded, again, by double quotes, thanks to the syntax : ("?)['.,]?(\w[\w '.-]*\w)['.,]?\1

On the other hand, it’s important to point pout that the duplicate name matched, with the regex ("?)['.,]?\2['.,]?\3 :

• Can have leading or trailing symbols, different from the first occurrence, to be deleted

• Can be surrounded, or not, with double quotes, independently, too, from the first occurrence, to be deleted

To end with, the names, with a single double-quote ( as "xxxx or yyyy" ) are considered as invalid entities. Indeed, let’s suppose the initial text, below :

{ 000 "123 555 456" 999 "123 456" 789 }

If names must be surrounded with double-quotes, or not, we get the same text, as there is no duplicate :

{ 000 "123 555 456" 999 "123 456" 789 }

If names as "123 or 456" were allowed, we would get the wrong text, below :

{ 000 555 999 "123 456" 789 }

So Marc, the new regex, below, should, correctly, miss very few names ;-)) And, thus, get rid of the great majority of the duplicates !

SEARCH (?si)(?:^|\h+)("?)['.,]?(\w[\w '.-]*\w)['.,]?\1(?=(?:\h+|\R)(?:[^{}]*(?:\h+|\R))?("?)['.,]?\2['.,]?\3(?:\h|\R))

And, with the free-spacing mode, which allows to identify the different parts of this regex, it gives :

SEARCH (?xsi) (?: ^|\h+) ("?) ['.,]? ( \w[\w '.-]*\w ) ['.,]? \1 (?= (?: \h+|\r?\n) (?: [^{}]* (?: \h+|\r?\n) )? ("?) ['.,]? \2 ['.,]? \3 (?: \h|\r?\n) )

Oh, I forgot to say that the search, is, from now on, insensitive to the case, due to the modifiers syntax (?is), at beginning of the regex. So, assuming the text :

{ ZhanYong Zhanyong }

The first word would, indeed, be a duplicate of the second one ans, thus deleted !

Best Regards,

guy038

Instances it failed.
__
“Ko cheng”

location, start of line after tabs, its twin was second from end of same line
__
This one i imagine will be a bit tricky :/ (if even possible)

ZhanYong,

Location anywhere, its twin has the Y lowercase instead.
__
'Abd

Location anywhere, culprit probably punctuation at start
__
cont.

Location anywhere, Probably reverse reason as above
__
Cheers. and Ive spread the one from last night to a few people, they pass their thanks to you for this. it saves so much time

Ah, I see ! So, I just changed the \x20+ syntax with the \h+ one, to include tabulations and No-Break space characters as possible separators

Secondly, in order to consider the apostrophe ' and the hyphen - as possible word character, I changed the syntax (\w[\w ]*\w) with the (\w[\w '-]*\w) one !

So, the final version of the regex is, from now on :

SEARCH (?s-i)(?:^|\h+)("?)(\w[\w '-]*\w)\1(?=(?:\h+|\R)(?:[^{}]*(?:\h+|\R))?("?)\2\3(?:\h|\R))

and, with the free-spacing mode, which allows to identify the different parts of this regex, it gives :

SEARCH (?xs-i) (?: ^|\h+) ("?) (\w[\w '-]*\w) \1 (?= (?: \h+|\r?\n) (?: [^{}]* (?: \h+|\r?\n) )? ("?) \2 \3 (?: \h|\r?\n) )

Just tell me if other characters, than the apostrophe, the hyphen and the space characters, may exist in your list of names :-))

Cheers

guy038

P.S. :

Note that, with the free-spacing regex (?x), the \R syntax is forbidden ! So, I changed it by the usual \r?\n syntax !

Only things its missing right now, are the start of lines, 2-3 tabs in (reference screenshot above) and ones with punctuation mid word. EG: Abu’l-Ghazi

But if its just 115 errors, i can handle that without much more work :D

Again i cant thank you enough, and ill be sharing this script with a fellow person having to deal with a very similar issue.

https://imgur.com/a/MO0dV

One hell of a job so far, this will get me able to finish this tonight likely (even with just this script part) myself and i very much appreciate the help.

I think that my new version should be very close to to your needs ! Just try it out :-)

So, assuming the following hypotheses :

• Names with one word of two word character, minimum or with several words, separated with, at least, one space character, possibly surrounded by double-quotes

• Names are preceded with, at least, one space char or are located at the beginning of a line

• Names are followed with , at least, one space char or are located at the end of a line

• Each block of names is embedded in a {.........} block, in a single line or split on several lines

• If a name is preceded of followed with a brace, one space character, at least, must separate them

Then, the correct version for removing duplicates names of each block, only, should be :

SEARCH (?s-i)(?:^|\x20+)("?)(\w[\w ]*\w)\1(?=(?:\x20+|\R)(?:[^{}]*(?:\x20+|\R))?("?)\2\3(?:\x20|\R))

Remark :

Compared to my previous version, this regex is more complex. Son for a best understanding, here is the equivalent version, with the free-spacing regex mode, which allows to insert non-significant space characters, in the regex !

SEARCH (?xs-i) (?: ^|\x20+) ("?) (\w[\w ]*\w) \1 (?= (?: \x20+|\r?\n) (?: [^{}]* (?: \x20+|\r?\n) )? ("?) \2 \3 (?: \x20|\r?\n) )

So given, for instance, the initial text below :

{
123 123 654
 321 456
 999 852 666 "852"
 123
}

{
 123 654 222 333 999
 852 "999" 456 123
 000 "123 654" "999"
}

{
 "123 654" 555 "111"
 852 111 "123" "000"
 999 "333" "123 654" 000 333
}

{
111 789
789
789 222
333 789 444
}
{
555 "789"
"789"
"789" 666
777 "789" 888
}

{            3456      "3456"      3456       }

{ "6789" 6789 "6789" }

{
 "456" "12 34 56" 456 "123" "456" 123 "12 34 56" 789
}

We get the following text :

{
 654
 321 456
 999 666 "852"
 123
}

{
 222 333
 852 456 123
 000 "123 654" "999"
}

{
 555
 852 111 "123"
 999 "123 654" 000 333
}

{
111

 222
333 789 444
}
{
555

 666
777 "789" 888
}

{      3456       }

{ "6789" }

{
 "456" 123 "12 34 56" 789
}

Waooooooo ! This regex totally drained me ;-))

Cheers,

guy038

I don’t give up :-)) Just making numerous tests. It should be OK, very soon…

guy038

EG
{
Line 1: 123 654
line 2: 321 456
}
{
Line 1: 123 654
Line 2: 582 456 123
}

Would only hit on the 123 pair in the second set

Ie
{
Line 1: 123 654
line 2: 321 456
}
{
Line 1: 654
Line 2: 582 456 123
}

Use this regex, below :

SEARCH (?s-i)\x20("?)(\w[\w ]*\w)\1(?=\x20(?:.+?\x20)?("?)\2\3\x20)

Compared to my previous one, I’ve just changed the part (?-is) by the syntax (?s-i)

So assuming your last example ( which needed some re-formatting, to be sure that names, with double quotes or not, are surrounded with space chars ! ), and with, BTW, 3 duplicates : 123, 321 and 852 !

{ 123 321 654 }
{ 123 951 753 "123" }
{ 456 852 "753 123" }
{ 123 "321" 852 }

AFTER replacement, we get the new text below :

{ 654 }
{ 951 753 }
{ 456 "753 123" }
{ 123 "321" 852 }

I hope, this last version is the good one ;-))

Best Regards

guy038

Example

Line 1 :123 321 654
Line 2 :123 951 753 “123”
Line 3: 456 852 “753 123”
Line 4: 123 “321” 852

123, 321 are both dupes in this example

I’m answering to myself ! For equivalent matches, I could simplify the regex to that shorter version :

SEARCH (?-is)\x20("?)(\w[\w ]*\w)\1(?=\x20(?:.+?\x20)?("?)\2\3\x20)

BTW, this new regex handle names with a minimum of two word characters ( Only one-letter names are not supported !! )

Cheers,

guy038

Perfect I’ve got the right regex to get what you want ! Just follow the few steps, below :

• First, of course, do a backup of your file !

• Open your file in N++

• Open the Replace dialog ( Ctrl + H )

• Type in the regex (?-is)\x20(?|"(\w[\w ]+\w)"|(\w[\w ]+\w))(?=\x20(.+?\x20)?("?)\1\3\x20) , in the Find what: zone

• Leave the Replace with: zone EMPTY

• Tick the Wrap around and the Regular expression options, ONLY

• Click on the Replace All button or, repeatedly, on the Replace button

Et voilà !

Remarks :

• I forgot to ask you about the case of the names : I mean, for instance, are the names Devi, DEVI, devi, DeVi,… all identical ? If so, just change the modifiers part (?-is), at beginning of the regex, into (?i-s)

• I assume that each block begins with an opening brace, followed with a space character and ends with a space character followed with a closing brace

• This regex correctly delete the first occurrence of two consecutive identical names, as below :

male_names = { "xxxxxx" "Barom Reachea" Barom Reachea "zzzzzz" }

• Note that it’s not very easy to visualize the entire gap between two duplicates, because of the overlapping phenomena, which may occur, like in :

male_names = { "xxxxxx" "Barom Reachea" "Soriyotei" Barom Reachea "yyyyyy" "Barom Reachea" "zzzzzz" Soriyotei }

So just use the Find dialog, with the regex above, which will match and select the first occurrence of duplicate names, before performing the Replace operation

Next time, Marc, I’ll give you some information on the regex itself, if you want to !

Best Regards,

guy038

P.S. :

May be, this complicated regex can be shortened ! Anyway, this one is working fine :-)

BEFORE : male_names = { “Ang Chan” “Barom Reachea” “Soriyotei” “Barom Reachea” “Sri Raja” Soriyotei }

AFTER : male_names = { “Ang Chan” “Barom Reachea” “Sri Raja” Soriyotei }

Marking for readability

{ |“Ang Chan”| |“Barom Reachea”| |“Soriyotei”| |“Barom Reachea”| |“Sri Raja”| |Soriyotei| }
{ |“Ang Chan”| |“Barom Reachea”| |“Sri Raja”| |Soriyotei| }

basically its read inside the | | marks if the inside of them is the same, then its a duplicate.There is only 1 space between each item. and in any case of duplication only one is kept
Eg 123 "123" 123 654 123 123 987 12
only one 123 would be kept

would need both the “Barom Reachea” extra and one of the two Soriyotei (preference leaning to quoted one) removed

I’m really sorry but I did not fully understand this sentence ! Do you mean that you want to delete the two occurrences of the "Barom Reachea" name, in your example ?

So, do you mind using that simple syntax, below, which clearly shows what you expect to ?

BEFORE : male_names = { "Ang Chan" "Barom Reachea" "Soriyotei" "Barom Reachea" "Sri Raja" Soriyotei }

AFTER : ???

On the other hand, what to do if there are more than 2 duplicates ? So, again, with the same syntax :

BEFORE : male_names = { "xxxxxx" "Barom Reachea" "Soriyotei" Barom Reachea "yyyyyy" "Barom Reachea" "zzzzzz" Soriyotei }

AFTER : ???

Finally, are the names, within double quotes or no, always separated with a single space character only ? Or is this kind of list possible ( with multiple spaces / tabulations, even 0 ), like below ?

male_names = { "xxxxxx"          "Barom Reachea" "Soriyotei" Barom Reachea	"yyyyyy" "Barom Reachea""zzzzzz" Soriyotei }

Just note, that according to the general behaviour of the N++ regex engine, the easiest way would be to delete all duplicates but the last one , whatever the name is surrounded, or not, with double quotes !

So, given the previous example :

male_names = { "xxxxxx" "Barom Reachea" "Soriyotei" Barom Reachea "yyyyyy" "Barom Reachea" "zzzzzz" Soriyotei }

we would obtain, AFTER replacement :

male_names = { "xxxxxx" "yyyyyy" "Barom Reachea" "zzzzzz" Soriyotei }

Cheers,

guy038

so

male_names = { "Ang Chan" "Barom Reachea" "Soriyotei" "Barom Reachea" "Sri Raja" Soriyotei }

would need both the “Barom Reachea” extra and one of the two Soriyotei (preference leaning to quoted one) removed

OK, Marc, I"ve already find out the right regex to match the gap between two duplicates names, of a same block !

Now, I need additional information :-)

Given, for instance, the text, below, with the Barom Reachea duplicate name :

male_names = { "Ang Chan" "Barom Reachea" "Soriyotei" "Barom Reachea" "Sri Raja" }

Do you expect, AFTER replacement, the text A :

male_names = { "Ang Chan" "Barom Reachea" "Soriyotei" "Sri Raja" }

OR the text B :

male_names = { "Ang Chan" "Soriyotei" "Barom Reachea" "Sri Raja" }

In addition, could you tell me which text cases may happen ? ( From your previous post, I understood that cases 1 and 2 do occur ! )

Case 1 : male_names = { "Ang Chan" "Barom Reachea" "Soriyotei" "Barom Reachea" "Sri Raja" }

Case 2 : male_names = { Ang Chan Barom Reachea Soriyotei Barom Reachea Sri Raja }

Case 3 : male_names = { "Ang Chan" "Barom Reachea" Soriyotei Barom Reachea "Sri Raja" }

Case 4 : male_names = { "Ang Chan" Barom Reachea Soriyotei "Barom Reachea" "Sri Raja" }

See you later,

Best Regards,

guy038

Also I wouldn’t mind if i had to say select each block 1 by one, i do have a external program that’s at least pointing out where each dupe is, but having to look by hand for each case is proving troublesome to keep focused