@gstavi said:
(You replied to my comment, but I’m not sure this was directed to me. In case it was…)
Professionals who search 100,000s of files with Notepad++ are not professionals.
Professionals, when they reach that scale (and long before), use command line tools. Then they open the found files in an editor. Possibly Notepad++.
I don’t really care who is or isn’t a “professional.” I started working on Search++ mostly to experiment with some user interface changes to provide more flexibility, while also wanting to expose the enhancements (formulas, more complete Unicode support) I built into the search in Columns++ in a more general-purpose framework.
It’s all way too much to propose as a change to the base program. I’m hoping that eventually (there is much work yet to be done) it can be a practical alternative to Notepad++’s built in search for people who like the user interface changes.
So, along that line, my current phase of development is working on Find in Files. Because I don’t rely on Scintilla’s interface for regex searching (I use a separate copy of Boost::regex with custom iterators directly against the Scintilla buffer), I can skip Scintilla entirely when reading files from a folder.
The main performance bottleneck of search in files was that every file was fully loaded into Notepad++, then its encoding was guessed. Probably with extra overhead as if this file is about to be displayed. Then its contents were scanned for matches as if it was regular search. Simple, elegant and SLOWWWW.
That part I skip, except that I still have to work out the encoding. Right now I handle all three forms of byte order mark, and in the absence of a BOM I differentiate valid UTF-8 (including pure ASCII) from anything else — anything else being processed using the system default code page encoding.
I don’t yet handle files with an internal character set declaration (like HTML or XML), UTF-16 without a byte order mark, or legacy code pages other than the system default.
The #1 required performance optimization is to have “search files as utf-8” feature which is basically grep. Assume all files are utf-8 (or ascii or binary). Scan them with efficient buffer by buffer linear algorithm without loading the entire file to Notepad++. But this is a different new feature, not optimization of current search which is still needed by people who (for some reason) use multiple encodings.
I have part of this optimization (no Scintilla and no encoding conversion, just use the correct iterator for the detected encoding), but I still read the entire file into memory, because processing the full possible range of Boost::regex regular expressions without the entire text present would be very challenging (if not impossible). I gather grep is normally limited to line-by-line processing; that wouldn’t fit with what people expect from regular expressions in the context of Notepad++.
It has occurred to me that a future (probably far future) enhancement to my plugin might be to offer the RE2 engine as an alternative. Since it doesn’t backtrack, processing gigantic files without needing to read too much data into memory at once would be possible.
The #2 performance optimization is probably asynchronous I/O before multithreading. Maybe with modern NVME drives this is less true than it used to be with spinning disks. Still true for search over network share.
I’m still experimenting. My last public version (Search++ 0.6.3.1) uses memory-mapped I/O for files that are not remote, but further testing has shown me that has little-to-no gain over just sequentially reading the files into a buffer for either my NVMe SSD or an internal hard drive.
The next thing I plan to try is “decoupling” reading and searching. Right now each thread reads and then searches; there might be some gain from separating those (a reading pool pushing to a queue from which a searching pool pulls), though my experience over the past two or three weeks has shown me I can spend a great deal of time pursuing a theoretically better alternative only to have it make no real-world difference at all.
So far the most stubborn problem I’m up against is figuring out how many I/O operations to submit at once. An SSD definitely benefits from letting all cores hit it with requests at once. A USB thumb drive suffers from that, and can actually perform worse than single-threaded… except when the files are cached from just having been read before, in which case they act like an SSD. For a remote share, the optimal number of simultaneous requests probably depends on the current state of the network — something I have no real way to model and test.
If you happen to know of a practical, real-world implementation of Windows code that determines at run time how many simultaneous read operations to issue against whatever is hosting a given folder for maximum throughput, please tell me about it.
Multithreading is 3rd priority. It is extremely unlikely with original algorithm and rather easy for search as utf-8.
I’ve managed to get around 13 times the performance of Notepad++ native search on an SSD in a system with an i-9 9900K (16 threads). Some of that is probably due to avoiding the use of Scintilla, and some (I think most) to multithreading.
I am definitely not a “professional” when it comes to having a clue how to measure performance properly, though, so not too much should be made of any of it.
But again, opinions of people who can’t mix and match the right tool for the job and insist that every tool should adapt itself to their selfish needs should not have much weight.
Agreed about the “insist” part. It’s almost always more productive to learn about what tools exist than to whine about how the one with which you are familiar doesn’t work the way you wish it did.
Yet users’ complaints are how those of us who develop software learn about what might be useful. That doesn’t mean we always can or should give people what they think they want — they know what impedes or annoys them, not the best way to make it better — but it tells us where existing tools fall short. It’s up to us to design ways to fill in the gaps, though not always the way a non-programmer might have imagined it.
