Code is not working

[Blake G. Barrington]

I am using Notepad++ 32-bit 8.6.7 on Windows 10.

I am trying to mark duplicate lines (not delete) in text documents. I am using Search Mode and Regular Expression, and have checked Bookmark Line and Mark All buttons.

First, I entered the expression: ^(.*?)$\s+(?=[\s\S]*?^\1$) under the Mark tab. It said, “Find invalid regular expression”.

Then, I tried: ^(.+?)\r?\n(?=\1) which found zero matches in entire file. I KNOW there is is at least one identical duplicate line.

Anyone knows the correct way of doing this?

-–

moderator added code markdown around text; please don’t forget to use the </> button to mark example text as “code” or ` characters around inline text (like `this` ⇒ this) so that characters don’t get changed by the forum

[PeterJones]

First, I entered the expression: ^(.*?)$\s+(?=[\s\S]*?^\1$) under the Mark tab. It said, “Find invalid regular expression”.

Hover over the ... button in the speach bubble on the error message. It will tell you that, essentially, the regex got too “complex”, because it has to keep backing up to try the next option. Basically, your file is too big.

See more in the User Manual on regex complexity

With a small file, your regex works just fine:

Then, I tried: ^(.+?)\r?\n(?=\1) which found zero matches in entire file. I KNOW there is is at least one identical duplicate line.

That one only finds duplicate lines if they are immediately next to each other:

vs

Anyone knows the correct way of doing this?

With very big files, it is non-trivial

• If the line order doesn’t matter, you could sort the file, then use your second regex.
• If you just want to delete the duplicates, then you can use Edit > Line Operations > Remove Duplicate Lines
• If you just want to mark them for later investigation, without changing line order, it’s going to be more difficult. I would probably recommend making two copies of the file, in addition to the original, as copy1 and copy2.
  • In copy1, I would just sort and run your second regex;
  • In copy2, I would
    • use Edit > Column editor to create initial line numbering (prefix with 0).
    • do a regex to move the number from the beginning to end of each line (^(?-s)(\d+)(?=\D)(.*)$ ⇒ $2\t$1
    • do the sort
    • do a regex to move the number to the end of each line (^(?-s)(.*?)\t(\d+)$ ⇒ $1\t$2
  • then you can do your second simpler duplicate search in copy1, and use copy2 to determine which line number each line was from originally

It’s not ideal, but without some magical regex manipulation to make it less backtracky (hints in the complexity section), you’re not likely to find a regex that doesn’t become invalid for you.

[Coises]

@PeterJones said:

It’s not ideal, but without some magical regex manipulation to make it less backtracky (hints in the complexity section), you’re not likely to find a regex that doesn’t become invalid for you.

@blake-g.-barrington , @PeterJones:

I can’t say for certain — much of Boost.regex (the regular expression engine Notepad++ uses) is still a black box to me — but I think it’s unlikely that any regex can do what is requested without getting the complexity error on large files. As best I can tell, Boost.regex is a bit “trigger happy” with that warning. It doesn’t really know whether the expression is trapped in a loop that isn’t progressing, it just knows that it seems to be scanning the same text over and over again. For any regex to solve this problem, it has to scan the same text over and over: for every line, it needs to check every following line up to the end of the file, unless it finds a match sooner. Given that and what I have observed about Boost.regex, I would not recommend spending a lot of time trying to improve the regular expression.

As @PeterJones noted, if you don’t need to preserve the order of lines in the files, the solution is simple: sort the file first.

If you do need to preserve order, the next question is whether this is a one-time thing you need to do to solve a specific problem, or a recurring task you’ll need to perform against various files in the future.

If it’s a recurring task, ditch the regular expressions and write a script. Python is probably the best language for scripting in Notepad++, but I’m not very familiar with Python, so I’ll leave it to someone else to explain that approach if that is what you need.

If this is a one time thing, then I would suggest:

• Insert line numbers at the beginning of each line.
• Make a zero-width rectangular selection following the line numbers and sort.
• Adapt your regular expression to find adjacent lines that contain duplicate text following the line number and make a replace expression that adds a marker character following the line number: a character that isn’t used otherwise in the file.
• Sort on the line numbers to put things back in order.
• Mark lines that contain the marker character.
• Use Replace All to delete the marker character.