Search++: A work in progress

Coises

@guy038 said:

I’m actually really surprised by Search++'s performance compared to Notepad++, especially when the search involves a complex regular expression. ( 0:16.36 vs 2:08.20 ! )

Please, note that the USB flash drive I used for my test is a fairly old one with a capacity of only 512 Mb, which may not be well-suited for multithreading and would explain the poor results with Search++

In any case, I assume you’re now confident in Search++'s overall performance !

I don’t entirely trust my own results. Why did my USB search in Search++ for the take longer than the search for a([^\r\n])[^\r\n]*b\1? Either the test data is erroneous or spurious, or Search++ is screwing up access to the USB in some way that’s causing it to “choke” and lose performance (what I suspect happened in your test). I’m still trying to work out how to avoid that happening without hurting the performance when reads are fast.

To the best of my knowledge, Notepad++ is entirely single-threaded. It’s only ever doing one thing at a time. An i9-9900K can run sixteen threads simultaneously. So for CPU-heavy workloads (complicated regex searches) with lots of files, it’s not at all surprising that a multithreaded approach could be almost eight times faster. Thirteen times faster running a([^\r\n])[^\r\n]*b\1 against an SSD is pretty satisfying, though. :-) I’m probably getting additional gains from using memory-mapped I/O and running Boost::regex directly against the data the operating system maps into memory, while Notepad++ loads every file into a buffer, and from there into an off-screen Scintilla, before running the search through Scintilla’s API instead of directly with Boost::regex.

It’s useful to have tests like yours. I won’t be able to test anywhere near all the situations that can occur. What happens when someone searches a OneDrive folder and not all the files are up-to-date? What happens when someone searches a drive that’s mounted over a VPN? What happens when the cat knocks the USB connector loose in the middle of a search? The worst case for all these is that the whole thing either crashes and takes Notepad++ down with it, or locks up and makes the user terminate Notepad++ with task manager — losing any unsaved work in progress. I don’t want to let that happen. I still have a lot to do to “harden” this against things that can go wrong.

And that’s not even starting with Replace yet.

sevem47

First of all thanks to your great work with this plugin. Going through the help file to learn about all features of the plugin the following ideas came to my mind.

They are about graphical topics that probably are just cosmetical questions.

Using a docking dialog for different plugins they may share the same space in NPP. If Search++ ist not active, there is just an empty space that is visible for me. Other plugins have a graphical representation that, in addition to the balloon tip, show which plugin is active. Two images to show my point:
/
Using Search++ I learned about the possibility to show only certain lines in the edit window. Would it make sense to have a marker that identifies that only some lines of a document are displayed after applying Search++ commands. Something like the following filter symbol to mark that not all lines are visible:

(the image is copied from Total Commander’s function ‘Synchronize directories’)

Just some ideas that would help me in using Search++.

Coises

@sevem47 said:

First of all thanks to your great work with this plugin.

Thank you for trying it and giving feedback!

Using a docking dialog for different plugins they may share the same space in NPP. If Search++ ist not active, there is just an empty space that is visible for me. Other plugins have a graphical representation that, in addition to the balloon tip, show which plugin is active.

Good point. I will fix that — not necessarily in the very next release, but I will fix it.

Using Search++ I learned about the possibility to show only certain lines in the edit window. Would it make sense to have a marker that identifies that only some lines of a document are displayed after applying Search++ commands.

I might be missing something. Do you find that it is not obvious enough in the document window itself when lines are hidden (the horizontal lines where the hidden lines are)?

sevem47

@Coises said:

I might be missing something. Do you find that it is not obvious enough in the document window itself when lines are hidden (the horizontal lines where the hidden lines are)?

Thanks for your hint; I see, the mistake is on my side! I use a different background color and my eyes are getting older ;-) Therefore I did not see the lines between the text:
My configuration vs. Default theme:
/
Now I know where to look.

Good point. I will fix that — not necessarily in the very next release, but I will fix it.

I know you have a lot of topics on your list that are much more urgent. For me it is a ‘nice to have’. Thanks for your help.

M Andre Z Eckenrode

This post is deleted!

M Andre Z Eckenrode

@Coises

As you know, I recently reported an issue with Search++ in a new, separate topic, but I see that several others have reported various issues in this one (presumably, because it’s still a work in progress). I have a new issue to report, and thought maybe I should ask if you prefer that new issues get reported in their own topic (which is generally my default), or for users to just include them in this one. Do you care one way or the other?

Coises

@M-Andre-Z-Eckenrode said:

@Coises

As you I recently reported an issue with Search++ in a new, separate topic, but I see that several others have reported various issues in this one (presumably, because it’s still a work in progress). I have a new issue to report, and thought maybe I should ask if you prefer that new issues get reported in their own topic (which is generally my default), or for users to just include them in this one. Do you care one way or the other?

Either way is OK with me. I suspect the moderators might prefer to keep discussion confined to this topic, though.

Coises

Search++ version 0.6.4 is available. It is expected to fix the bug reported here.

Even though it seems most people never experience that bug, I still recommend upgrading at the next convenient time, so we will all be testing the same code. It’s always possible that I broke something else.

I am still working on improving the multi-threading for find in files. I’m also trying to decide how best to mitigate a problem wherein Search++ fails to load when a particular Windows redistributable library has never been installed on the system. More updates should be coming before too much longer.

All thoughts, suggestions, comments, criticisms, bug reports and raspberries are welcome, and thank you to everyone willing to give this jalopy a good work-out.

Coises

Search++ version 0.6.5 is available:

Update to ICU 78.3 and use static linking for ICU. This is expected to fix problems some testers have had running Search++ on older systems or on minimal Windows installs (like Windows Sandbox). For most users there will be no functional difference between this version and version 0.6.4.

Note: There are five icu——.dll files in older versions of Search++ that are not used beginning with version 0.6.5; so it is best to delete the existing Search++ folder (or its contents) before copying, rather than copying the new folder over the old one.

Coises

@guy038 said:

Now, I did a test, searching in all files of my D: USB key, with :
The native N++ Find in Files dialog

The Search++ Search in Files dialog

The MultiReplace Find All in Files command
To be sure to process fair tests, I unplung my USB key, re-start my laptop completely, replug my USB key and starts a new N++ session, before running these three Find in Files operations, in turn !

I searched for the Fi string, with the Match case and the Regular expression options checked. I got :
27,112 matches in 833 files of 1,761 searched for the N++ Search Results panel, in 1m 37s

27,112 matches in 833 files of 1,761 files for the Search++ Search++ Results, in 2m 21s

27,087 hits in 835 file(s) for the MultiReplace Search results panel, in 1m 36s

I have been struggling with how to optimize the multithreading of my search for over a month now. Among other things, I’ve found that most everything I thought I was measuring wasn’t what I was actually measuring.

One surprise I only recently discovered was that even if I eject and unplug a thumb drive and then plug it in again. there is still a “first time today” cost in the time it takes to read it. Windows is persisting something and it is able to identify the same drive, even though it’s been unplugged. I don’t know the exact time, but Windows on my machine keeps it for more than an hour and less than 24 hours.

Running a particular test the first time I plugged in a particular USB drive I have, it took 8 minutes and 26 seconds. After ejecting the drive, unplugging it, waiting a moment, plugging it back in and running the exact same test, it took 20 seconds.

This drive appears to take whatever it is that Windows does once a day (?) exceptionally hard. Most drives don’t show anything so obvious. This is the only drive I have tested so far that uses FAT, so that could have something to do with it.

I haven’t yet tested whether rebooting is enough to reset the “once a day” penalty, or if it persists in something that isn’t lost in a reboot. It may have something to do with antivirus, though I haven’t yet proven any such thing.

So I’m wondering, if you happen to remember, whether you tested in the order you listed, or whether you happened to test Search++ first?

In any case, after weeks of running in circles, I think I will soon give up the fight to optimize performance and see what else needs work.