Improving Find in Files performance

Coises

@guy038 said:

Unfortunately, I must add, that, for four tests above, the resulting Results panel, after the search phase, was rather unusable and did not displayed correctly as really too large !

In what sense did it not display correctly? Did it fail to load at all? Was the information corrupt, or some data missing? Was the behavior roughly the same for Notepad++/Vitalii-Notepad++ and for Search++?

II tried one last round of tests, searching every pdf file within my SSD drive of 512 Gb, with the same parameters than above.

Using the same protocol as above, here are the results :

•----------------•--------------------------------------------•------------•------------•----------------------------•
|    Program     |                 Results                    |  Scan ...  |   Total    |           Errors           |
•----------------•--------------------------------------------•------------•------------•----------------------------•
|  N++ Vitalii   |  74,854 hits in 130 files of 130 searched  |  5 m 20 s  |  5 m 24 s  |                            |
•----------------•--------------------------------------------•------------•------------•----------------------------•
|  Native N++    |  74,854 hits in 130 files of 130 searched  |  5 m 29 s  |  5 m 32 s  |                            |
•----------------•--------------------------------------------•------------•------------•----------------------------•
|  Search++      |  74,854 matches in 130 documents           |  3 m 58 s  |  4 m       |                            |
•----------------•--------------------------------------------•------------•------------•----------------------------•
|  MultiReplace  |  ERROR after 3,6s and about 116,000 files  |     3,6 s  |    n/a     |  Error IDENTICAL to above  |
•----------------•--------------------------------------------•------------•-----------------------------------------•

What did I observe :

The Vitalii Dovgan version is slightly faster than native N++
Again, due to multi-thread processing, the Search++ plugin is the winner

What I find curious is that most of the difference is in the scanning phase. The directory scan in Search++ is not multi-threaded, though it does run in a background thread. So I don’t know what it’s doing that’s faster than Notepad++ native. I did code it from scratch, not even looking at Notepad++ code, so it might not use the same APIs.

If the total number of scanned files is SMALL, any program will do. Paradoxically, the multithreaded process seems to take longer

I’m fairly sure that the difference isn’t the number of files scanned, it’s the device. There is something about the way I process files that does not work well with your USB drive. I tried to reproduce this with devices I have over the several weeks I spent trying to optimize the multi-threading, but I was never able to find anything that behaved the same way as yours.

I’m sure there are other devices out there that will show the same behavior. I just don’t know how to diagnose and fix the problem when I can’t reproduce it.

Thomas Knoefel

@Coises said:

Again, due to multi-thread processing, the Search++ plugin is the winner

I insist! MultiReplace is the fastest, done in just seconds! Just ignore the bug ;-) I’ll have a fix ready soon…

Vitalii Dovgan

I gave a try to your Windows Release-x64 version of N++ v6.9.8 from https://github.com/notepad-plus-plus/notepad-plus-plus/pull/18376

That’s interesting!
Were you searching or replacing?
The replacing performance should be very close since almost the same code is executed.
The searching (find only) performance is expected to be 2-3 times faster.

Vitalii Dovgan

Here are my average results:

Search "text" (122307 hits in 2161 files of 4385 searched)

standard Notepad++  -  00:15.37
modified Notepad++  -  00:08.15

Search "cache" (19176 hits in 787 files of 12821 searched)

standard Notepad++  -  01:00.02
modified Notepad++  -  00:33.54

Search "cache" (45120 hits in 2316 files of 18920 searched)

standard Notepad++  -  02:53.74
modified Notepad++  -  02:12.04

The modified Notepad++ demonstrates better performance on text files rather than binary files.
That’s why my very last search, which included many binary files, is not 2 times faster.

To see the full power of the modified Notepad++, apply file masks that will exclude most of binary files.

Vitalii Dovgan

Here are results for the same folder for mostly text files (the Filters are: *.* !*.exe !*.dll !*.bin !*.cab !*.msi !*.sys) :

Search "cache" (17205 hits in 343 files of 11839 searched)

standard Notepad++  -  00:44.52
modified Notepad++  -  00:22.15

guy038

Hi, @vitalii-dovgan, @coises, @thomas-knoefel and All,

@vitalii-dovgan, you said :

Were you searching or replacing?

I just did Find in Files actions. I didn’t intend to modify the files on my SSD. Of course, I could had run a Replace in Files action by searching the Fi string and replace with that same string !. But I did not run this kind of test.

@thomas-knoefel, I sure that you’ll correct this bug, very soon !!

@Coises, you said :

In what sense did it not display correctly? Did it fail to load at all? Was the information corrupt, or some data missing? Was the behavior roughly the same for Notepad++/Vitalii-Notepad++ and for Search++?

I re-ran my previous search on any file of my SSD drive C: with Search++ :

2 m 26 s after the beginning, the scanning phase was over and the file list began to fill up
10 m 01 s later, the waiting files number was about equal to the finished files number !
3 m 28 s later, the files list was completed
10 s later, the Search++ Results panel was displayed, as shown in the snapshot below :

Note that, in this first snapshot, the docked Search++ dialog and the Search in Files dialog seem to be already inaccessible !
Then, I saw, in the N++ title, that the program did not respond anymore. Then, trying to stop Notepad++, I choose the answer Wait till the program responds
I then saw, this snapshot :

Clicking twice, on the OK button, I got, successively, these two snapshots :

Finally, after clicking OK in this last dialog box, it has disappeared.
Unfortunately, clicking again on the Search++ results, closed N++ completely

Best Regards,

guy038

Remainder :

In this second test of the scan of all files of C:\, the Search++ results were : 9,550,124 matches in 90,532 of 316,006 files, with 583 errors.

Coises

@guy038 said:

Scintilla error: status code 2

Thank you for the explanation and screen shots. Scintilla status code 2 means “Memory is exhausted” — so just as you suggested, the results list was simply too big for Scintilla to handle on your machine.

This is still a good point to bring up: it means I should at least experiment and see if I can catch that error before it propagates to killing Notepad++. It might not be possible, but I won’t know until I try.

guy038

Hi, @coises,

Keep in mind that I still have 32 Gb of RAM ! I know, it’s never enough :-((

BR

guy038

Thomas Knoefel

@guy038 said:

@thomas-knoefel, I sure that you’ll correct this bug, very soon !!

Yeah, maybe sooner than expected! Here is the fixed version of MultiReplace. Feel free to test the “Find in Files” performance and compare it with the other searches:
Here is the fixed version

guy038

Hello, @thomas-knoefel, @vitalii-dovgan, @coises and All,

@thomas-knoefel, I tested your last release and I’m pleased to confirm that no errors ocurred anymore during all the tests ! I followed the same protocol, that I described in my initial post.

Here are the results when scanning all the files of my SSD Drive C:\ :

•----------------•----------------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------------------------------------------------------•
|    Program     |                        Results                                             |  Scan Phase  |  Search End  |  Results panel  |                                   Details                                    |
•----------------•----------------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------------------------------------------------------•
|  MultiReplace  |  Search "Fi" (9 991 249 hits in 90 330 file(s)) [315 503 file(s) searched  |    5 m 56 s  |   48 m 15 s  |      48 m 27 s  |  1 115 skipped: 1 111 unreadable, 4 not decodable; 82 folder(s) unreadable]  |
•----------------•----------------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------------------------------------------------------•

But the nice thing is that your Search results panel was correctly displayed and functional :

Starting with the default Unfoll All option, I got these successive results :

•------------------------•----------•
|        Action          |   Time   |  
•------------------------•----------•
|  Collapse to File List |   32 s   |
|                        |          |
|  Fold All              |   24 s   |
|                        |          |
|  Unfold All            |   0,8 s  |
|                        |          |
|  Fold All              |  32,5 s  |
•------------------------•----------•

On the other hand, I went to the last file of the list, opened it and double_clicked on the first occurrence, in line 690. At once, the corresponding file was displayed in N++ with the caret on this first occurrence, in line 690 !

Now, here are the results when scanning all PDF files of my SSD drive C:\

•----------------•------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------•
|    Program     |                       Results                                    |  Scan Phase  |  Search End  |  Results panel  |           Details            |
•----------------•------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------•
|  MultiReplace  |  Search "Fi" (47 854 hits in 130 file(s)) [130 file(s) searched  |    5 m 45 s  |    5 m 48 s  |       5 m 48 s  |  ; 82 folder(s) unreadable]  |
•----------------•------------------------------------------------------------------•--------------•--------------•-----------------•------------------------------•

Note that you find the same results as with the other Search programs, in my initial post : that’s quite reassuring !

Best Regards,

guy038