How to find and remove duplicate strings of alphanumeric characters from multiple files?

Ramanand Jhingade

@Terry-R @guy038 Let me be more specific. I want to check for matter between ```
<

and


and replace any duplicated matter but keep the  original. I don't mind using the "Replace" function  instead of the "Replace all" function so that I can check what happens each time

Ramanand Jhingade

This post is deleted!

Ramanand Jhingade

@guy038 @Terry-R Sorry, I was unable to post it correctly. It should find and replace duplicate matter between “<” and “>” and keep the original

Terry R

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

but it is removing even the brackets in the CSS section of my .html files/pages (CSS can have just one bracket in a line). Any better suggestion?

You have yet to supply any examples. Regardless I will give you some help.

Since you are using the “Replace” button to see changes 1 at a time, when it gets to a line you don’t want to change press the “Find Next” button instead as this means you don’t change what is currently selected, instead it finds the next occurrence.

It might be possible to adjust the regex to ignore lines based on the characters on it but that needs examples provided.

Terry

Ramanand Jhingade

@Terry-R Suppose I want to avoid removing the flower brackets,

and

how to do so?

PeterJones

@Ramanand-Jhingade ,

As has been explained to you before, this process works best if you show two black boxes, one with the way your data is now (“have”); and the second with the way you want the data to look (“want”)

have:

blah
{
blah
}

want:

something
<
yada
>

Since you don’t seem to remember this guidance, I will re-post it. If you don’t follow these guidelines, you will find it hard to convince people to help you, especially since you’re asking so many questions.

----

Do you want regex search/replace help? Then please be patient and polite, show some effort, and be willing to learn; answer questions and requests for clarification that are made of you. All example text should be marked as literal text using the </> toolbar button or manual Markdown syntax. To make regex in red (and so they keep their special characters like *), use backticks, like `^.*?blah.*?\z`. Screenshots can be pasted from the clipboard to your post using Ctrl+V to show graphical items, but any text should be included as literal text in your post so we can easily copy/paste your data. Show the data you have and the text you want to get from that data; include examples of things that should match and be transformed, and things that don’t match and should be left alone; show edge cases and make sure you examples are as varied as your real data. Show the regex you already tried, and why you thought it should work; tell us what’s wrong with what you do get. Read the official NPP Searching / Regex docs and the forum’s Regular Expression FAQ. If you follow these guidelines, you’re much more likely to get helpful replies that solve your problem in the shortest number of tries.

Ramanand Jhingade

@PeterJones @guy038 I would like all the matter/characters between the < and > which have duplicates to be removed.
For example,

<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">
<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">

should become:-

<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">```

Terry R

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

I would like all the matter/characters between the < and > which have duplicates to be removed.

Thank you for finally giving us some examples. From those examples I can say that the original solution I linked from @guy038 along with his subsequent line numbering and reversing should do that. But I think you probably know that already.

What is missing is examples of the Suppose I want to avoid removing the flower brackets, question you asked earlier. Can you provide examples of the lines they are in or is it that these brackets are on lines by themselves, possibly with spaces before the bracket? This is very important information.

Terry

Ramanand Jhingade

@Terry-R In the CSS section of my webpages (just after my meta tags right at the top), some lines are just { or } and I don’t want those to be removed or replaced

Alan Kilborn

@Ramanand-Jhingade said:

I would like all the matter/characters between the < and > which have duplicates to be removed.
For example…

@Terry-R said

From those examples I can say that the original solution I linked … along with … subsequent line numbering and reversing should do that.

I’m confused why this command was not recommended for this task, given the example data provided:

Terry R

@Alan-Kilborn OP wants to use “Replace” in single mode, presumably to verify each removal. And considering he also has the {} characters to contend with I don’t blame him.

Terry

Terry R

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

some lines are just { or } and I don’t want those to be removed or replaced

As you never showed me examples of these lines I have to assume some things.

Consider adding ^(?=[^\{\}]+$) to the regex immediately behind the (?-is) if that’s what you have. This means look forward at the line and make sure there are NO { or } characters on it. If none then process the line to check if a duplicate.

Terry

Ramanand Jhingade

@Terry-R Thanks. I don’t want to mess up things, so please clarify if I should use (?-is)^(?=[^\{\}]+$)^\d+\h+(.+\R)(?=(?is:.*)^\d+\h+\1) in the Regular expression mode with the “new line” checked/ticked and click on the Find button?
If everything is done correctly, can I use Replace all for all files of the folder. @Alan-Kilborn says that searchingbackwards will not give accurate results in the post you linked to, but since we are reversing the order of the lines and searching, that should not be a problem, right (it will not be searching backwards)?

Terry R

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

Thanks. I don’t want to mess up things, so please clarify if

So assuming you are adding line numbers as @guy038 post mentioned in the link I provided and then sorting numerically descending the lines, then the search will be a forward search. Yes a backwards search is problematic but you are not doing that.
So the regex should be
(?-is)^(?=[^\{\}]+?$)\d+\h+(.+\R)(?=(?s:.*)^\d+\h+\1)

I did make 3 amendments:

We don’t need to include ^ twice.
I included a ? in my latest addition, this should prevent it taking too long if there aren’t any more { or } in the remainder of the file. Previously it would have searched multiple lines. The outcome is still the same but it was an omission I’d prefer to fix.
Remove the i you added near the end of the regex. This changes the search to a case insensitive one. You wanted it to be sensitive (h not equal H), not insensitive (h = H).

As for using “Replace in All Files” I would not do that until you have fully tested a number of files. Complicated regexes as this may have an effect on a line that you did not advise us about and that you did NOT want removed. I refer to the { and } lines which you did NOT originally mention, there may be others.

Open a file, complete the steps @guy038 mentioned for adding line numbers and sorting, then use the “Find” button first. If you get a line which you don’t want to remove, press the “Find Next” button NOT the “Replace” button. If the line can be removed, press the “Replace” button. Continue though the file until completed. For any lines which you don’t want to remove, you need to copy those elsewhere and then after finishing show us all of them. Only by doing that can we possibly amend the regex to ignore those lines as well, just like I did for the { and } lines.

Good luck
Terry

Terry R

@Terry-R said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

As for using “Replace in All Files” I would not do that until you have fully tested a number of files.

@Ramanand-Jhingade
On further reflection I realize that you will not be able to use the “Find in all Files”. The reason is that you need to add line numbers and sort before using the regex. Then afterwards you need to sort back to the original order and lastly remove the line numbers.

Most of those steps must be completed on an open file. On a positive note, these steps can be recorded as a macro. The steps would be (once macro recorded).

Open all the files
Run the macro. This macro moves cursor to first position, adds line numbers (with a possible blank between numbers and original line), sorts descending. Then it saves and closes the file. You can get a macro to run xx times where xx is the number of open files.
Run the regex using “find in files” option.
Re-open all the files and run a 2nd macro which sorts back to original order and removes the line numbers. Then it saves and closes the files by running it xx times as for step 2.

These steps will only be of an advantage if this is a function you need to carry out on a regular basis. If you are just fixing a problem once then I suggest opening each file one at a time and perform the steps.

Terry

Ramanand Jhingade

@Terry-R I thought you may be waiting for feedback. On using the code you gave, I got a Can't find the text (?-is)^(?=[^\{\}]+?$)\d+\h+(.+\R)(?=(?s:.*)^\d+\h+\1) message, so I clicked on the link you gave and used the code mentioned by @guy038 and searched for duplicate lines one by one. I then observed that only the 2 lines I typed above and their duplicates were a problem, so I removed all 4 lines. I then used the next line which happened to be unique and added the 2 lines I mentioned above using the extended mode with line1\r\nline2\unique line and added those 2lines in all the files of the folder. I thank you for your time and help. I also thank @PeterJones and @Alan-Kilborn for their inputs and @guy038 for the code!

Robin Cruise

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

<link rel=“stylesheet” type=“text/css” href=“engine1/style.css” media=“screen”>
<link href=“css/style.css” rel=“stylesheet” type=“text/css” media=“all”>
<link rel=“stylesheet” type=“text/css” href=“engine1/style.css” media=“screen”>
<link href=“css/style.css” rel=“stylesheet” type=“text/css” media=“all”>

This will find any duplicate line, but also the blank ones:

Search: (?-s)^(.*)\R(?s)(?=.*^\1(?:\R|\z))

Robin Cruise

@Ramanand-Jhingade said in How to find and remove duplicate strings of alphanumeric characters from multiple files?:

@PeterJones @guy038 I would like all the matter/characters between the < and > which have duplicates to be removed.
For example,

<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">
<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">

should become:-

<link rel="stylesheet" type="text/css" href="engine1/style.css" media="screen">
<link href="css/style.css" rel="stylesheet" type="text/css" media="all">```

I believe the solution will be:

FIND: (^<link.+)\R(?=[\s\S]*\1)
Replace by: LEAVE EMPTY

Ramanand Jhingade

@Robin-Cruise Thank you. I already did the needful!