Search++: A work in progress

Coises

Fix a pair of errors that caused incorrect results when characters outside the basic multilingual plane were used in a regex replacement string and when a capture group included invalid UTF-8.
Fix problematic behavior for Show file in explorer on the context menu for the file list in the Search in Files dialog when the file list is large. The action could become sluggish or fail to complete and interfere with other activity.
Handle absolute file paths larger than MAX_PATH in Search in Files. (They still won’t open in Notepad++, but there was no reason Search in Files shouldn’t be able to parse them.)
Avoid “configuration file for this plugin was edited” when running multiple instances of Notepad++: only save when the last instance is closing.

Nothing exciting here, and hopefully no new bugs… just some particular fixes.

Apologies to anyone who downloaded 0.6.3 in the short time it was available. I chose a dumb way to detect the last running instance of Notepad++ that was bound to cause a problem sooner or later. In 0.6.3.1 it is replaced with a less dumb way.

Coises

@guy038 said:

Now, based on my test, have you conducted any similar tests of your own, and have you found that searching with Search++ also takes longer compared to the native search in N++ ?

I ran some tests this morning because I wanted to get a reference before I start testing changes to the multithread scheduling in Search++ — so I can verify that what is supposed to be better actually is better.

This is Windows 10 Pro, 64 bit, running on a 2019 desktop with a Core i9-9900K and 32 GB RAM.
D is an internal SSD. S is an internal HDD. U is a USB thumb drive. \\PEACOCK is another machine on the same local area network.
The content of the folder Testing2 on each drive is the same. It’s a mix of 3,762 files in 228 folders, 591 MB total. Somewhere around 5% are binary files (dll, exe, png, etc.).
For U, following your lead, I did a “repeat” running the same search immediately after the first; after that, I unplugged and reconnected the drive before proceeding to another test.
I couldn’t conveniently unplug the internal drives, but testing showed no significant variation in successive trials on D, S or PEACOCK.

Some of these results are surprising. (I thought the hard disk would be noticeably slower than the SSD, and I expected to see a repeat of your experience with the USB drive.) I suspect that some specialized knowledge I do not have, and perhaps a more controlled environment, is required to measure performance meaningfully, so all this should be taken as anecdotal.

Search++ 0.6.3.1:

D:\Quick\Testing2     a([^\r\n])[^\r\n]*b\1  0:08.71  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
D:\Quick\Testing2     the                    0:01.44  445,907 matches in 3,342 of 3,762 files (Regex): the
S:\Testing2           a([^\r\n])[^\r\n]*b\1  0:08.91  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
S:\Testing2           the                    0:01.10  445,907 matches in 3,342 of 3,762 files (Regex): the
U:\Testing2           a([^\r\n])[^\r\n]*b\1  0:16.36  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
U:\Testing2 (repeat)  a([^\r\n])[^\r\n]*b\1  0:08.87  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
U:\Testing2           the                    0:24.84  445,907 matches in 3,342 of 3,762 files (Regex): the
U:\Testing2 (repeat)  the                    0:01.03  445,907 matches in 3,342 of 3,762 files (Regex): the
\\PEACOCK\Testing2    a([^\r\n])[^\r\n]*b\1  0:12.32  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
\\PEACOCK\Testing2    the                    0:07.07  445,907 matches in 3,342 of 3,762 files (Regex): the

Notepad++ 8.9.7:

D:\Quick\Testing2     a([^\r\n])[^\r\n]*b\1  1:53:64  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
D:\Quick\Testing2     the                    0:14.14  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]
S:\Testing2           a([^\r\n])[^\r\n]*b\1  1:53.99  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
S:\Testing2           the                    0:12.42  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]
U:\Testing2           a([^\r\n])[^\r\n]*b\1  2:08.20  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
U:\Testing2 (repeat)  a([^\r\n])[^\r\n]*b\1  1:54.96  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
U:\Testing2           the                    0:31.62  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]
U:\Testing2 (repeat)  the                    0:14.16  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]
\\PEACOCK\Testing2    a([^\r\n])[^\r\n]*b\1  2:11.36  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
\\PEACOCK\Testing2    the                    0:29.46  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]

(The reason for using [^\r\n] instead of . is that Notepad++ and Search++ have slightly different definitions of . when “matches newline” is not in effect: like Columns++, Search++ excludes only the characters that cause a line break in Scintilla, new line and return; while Notepad++ also excludes four other control characters, FF, NEL, LS and PS. The . is noticeably faster, but it returns different results for Search++ and Notepad++, almost certainly due to the binary files in Testing2.)

Edit: Added local area network tests.

guy038

Hello, @coises and All,

Your tests are much more comprehensive than my simple test and yield some interesting results !

In particular, the comparison between Notepad++ and Search++ for searches on a USB drive shows that Search++ outperforms Notepad++, whether the search is complex or simple !

Indeed, we have :

Search++ 0.6.3.1 :

U:\Testing2           a([^\r\n])[^\r\n]*b\1  0:16.36  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1
U:\Testing2 (repeat)  a([^\r\n])[^\r\n]*b\1  0:08.87  149,919 matches in 2,505 of 3,762 files (Regex): a([^\r\n])[^\r\n]*b\1

U:\Testing2           the                    0:24.84  445,907 matches in 3,342 of 3,762 files (Regex): the
U:\Testing2 (repeat)  the                    0:01.03  445,907 matches in 3,342 of 3,762 files (Regex): the

And :

Notepad++ 8.9.7 :

U:\Testing2           a([^\r\n])[^\r\n]*b\1  2:08.20  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]
U:\Testing2 (repeat)  a([^\r\n])[^\r\n]*b\1  1:54.96  Search "a([^\r\n])[^\r\n]*b\1" (149919 hits in 2505 files of 3762 searched) [RegEx]

U:\Testing2           the                    0:31.62  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]
U:\Testing2 (repeat)  the                    0:14.16  Search "the" (445907 hits in 3342 files of 3762 searched) [RegEx]

I’m actually really surprised by Search++'s performance compared to Notepad++, especially when the search involves a complex regular expression. ( 0:16.36 vs 2:08.20 ! )

Please, note that the USB flash drive I used for my test is a fairly old one with a capacity of only 512 Mb, which may not be well-suited for multithreading and would explain the poor results with Search++

In any case, I assume you’re now confident in Search++'s overall performance !

Best Regards,

guy038

Coises

@guy038 said:

I’m actually really surprised by Search++'s performance compared to Notepad++, especially when the search involves a complex regular expression. ( 0:16.36 vs 2:08.20 ! )

Please, note that the USB flash drive I used for my test is a fairly old one with a capacity of only 512 Mb, which may not be well-suited for multithreading and would explain the poor results with Search++

In any case, I assume you’re now confident in Search++'s overall performance !

I don’t entirely trust my own results. Why did my USB search in Search++ for the take longer than the search for a([^\r\n])[^\r\n]*b\1? Either the test data is erroneous or spurious, or Search++ is screwing up access to the USB in some way that’s causing it to “choke” and lose performance (what I suspect happened in your test). I’m still trying to work out how to avoid that happening without hurting the performance when reads are fast.

To the best of my knowledge, Notepad++ is entirely single-threaded. It’s only ever doing one thing at a time. An i9-9900K can run sixteen threads simultaneously. So for CPU-heavy workloads (complicated regex searches) with lots of files, it’s not at all surprising that a multithreaded approach could be almost eight times faster. Thirteen times faster running a([^\r\n])[^\r\n]*b\1 against an SSD is pretty satisfying, though. :-) I’m probably getting additional gains from using memory-mapped I/O and running Boost::regex directly against the data the operating system maps into memory, while Notepad++ loads every file into a buffer, and from there into an off-screen Scintilla, before running the search through Scintilla’s API instead of directly with Boost::regex.

It’s useful to have tests like yours. I won’t be able to test anywhere near all the situations that can occur. What happens when someone searches a OneDrive folder and not all the files are up-to-date? What happens when someone searches a drive that’s mounted over a VPN? What happens when the cat knocks the USB connector loose in the middle of a search? The worst case for all these is that the whole thing either crashes and takes Notepad++ down with it, or locks up and makes the user terminate Notepad++ with task manager — losing any unsaved work in progress. I don’t want to let that happen. I still have a lot to do to “harden” this against things that can go wrong.

And that’s not even starting with Replace yet.

sevem47

First of all thanks to your great work with this plugin. Going through the help file to learn about all features of the plugin the following ideas came to my mind.

They are about graphical topics that probably are just cosmetical questions.

Using a docking dialog for different plugins they may share the same space in NPP. If Search++ ist not active, there is just an empty space that is visible for me. Other plugins have a graphical representation that, in addition to the balloon tip, show which plugin is active. Two images to show my point:
/
Using Search++ I learned about the possibility to show only certain lines in the edit window. Would it make sense to have a marker that identifies that only some lines of a document are displayed after applying Search++ commands. Something like the following filter symbol to mark that not all lines are visible:

(the image is copied from Total Commander’s function ‘Synchronize directories’)

Just some ideas that would help me in using Search++.

Coises

@sevem47 said:

First of all thanks to your great work with this plugin.

Thank you for trying it and giving feedback!

Using a docking dialog for different plugins they may share the same space in NPP. If Search++ ist not active, there is just an empty space that is visible for me. Other plugins have a graphical representation that, in addition to the balloon tip, show which plugin is active.

Good point. I will fix that — not necessarily in the very next release, but I will fix it.

Using Search++ I learned about the possibility to show only certain lines in the edit window. Would it make sense to have a marker that identifies that only some lines of a document are displayed after applying Search++ commands.

I might be missing something. Do you find that it is not obvious enough in the document window itself when lines are hidden (the horizontal lines where the hidden lines are)?

sevem47

@Coises said:

I might be missing something. Do you find that it is not obvious enough in the document window itself when lines are hidden (the horizontal lines where the hidden lines are)?

Thanks for your hint; I see, the mistake is on my side! I use a different background color and my eyes are getting older ;-) Therefore I did not see the lines between the text:
My configuration vs. Default theme:
/
Now I know where to look.

Good point. I will fix that — not necessarily in the very next release, but I will fix it.

I know you have a lot of topics on your list that are much more urgent. For me it is a ‘nice to have’. Thanks for your help.

M Andre Z Eckenrode

This post is deleted!

M Andre Z Eckenrode

@Coises

As you know, I recently reported an issue with Search++ in a new, separate topic, but I see that several others have reported various issues in this one (presumably, because it’s still a work in progress). I have a new issue to report, and thought maybe I should ask if you prefer that new issues get reported in their own topic (which is generally my default), or for users to just include them in this one. Do you care one way or the other?

Coises

@M-Andre-Z-Eckenrode said:

@Coises

As you I recently reported an issue with Search++ in a new, separate topic, but I see that several others have reported various issues in this one (presumably, because it’s still a work in progress). I have a new issue to report, and thought maybe I should ask if you prefer that new issues get reported in their own topic (which is generally my default), or for users to just include them in this one. Do you care one way or the other?

Either way is OK with me. I suspect the moderators might prefer to keep discussion confined to this topic, though.