@guy038 said:
I understood the small differences which occur in the MultiReplace search ( 27,112 - 25 ) matches in ( 833 + 2 ) files. But, I prefer, first, to study more closely the MultiReplace plugin, which is very powerful too and reply later to @thomas-knoefel. It concerns the encoding of 4 files which is not recognized properly and which occurs only when the Find in Files command is used. The Find All and Find in Docs commands, themselves, give correct results for these 4 files !?
As best I can make out, like Search++, MultiReplace processes searches against documents open in Notepad++ using the copy of the document that Notepad++ has loaded into Scintilla. (I can’t really think of another way it could be done.) That means that for existing files, Notepad++ has already determined the encoding and either loaded it as ANSI or UTF-8, or translated it to UTF-8. (Notepad++ never sets Scintilla to use any encoding other than UTF-8 or the system default code page; anything else is translated to UTF-8 and translated back on writing.)
So it’s no surprise that we’re both using the same encoding as Notepad++ for Find All and Find in Open Documents, because we’re just using the work Notepad++ has already done.
I think MultiReplace uses an invisible Scintilla to process searches in files. It could be that it hasn’t entirely replicated the labyrinthine logic Notepad++ uses to determine the encoding of a file (including the effect of Preferences | MISC. | Autodetect character encoding).
Search++ does not duplicate that logic either, but it approaches the problem in a different way. Rather than using an invisible Scintilla control, Search++ uses its customized version of Boost::regex directly on the data in the file buffer. So I never translate the file encoding to anything else. I do have to determine the encoding, though, as the choice of iterator for Boost::regex depends on that.
There are limitations, not yet formally documented, to the way Search++ handles encodings for Search in Files:
UTF-16 files without a byte order mark are not recognized as such. (They’ll be misread as something else.)
UTF-8 files without a byte order mark that contain any invalid UTF-8 sequences will be processed using the system default (ANSI) code page.
Pure ASCII files are processed as UTF-8. (This doesn’t matter for Find, but I will have to review it when I implement Replace, since someone might include non-ASCII characters in the replacement text. I think there will have to be a user control to determine whether to promote ASCII to ANSI or to UTF-8.)
Files without a byte order mark that are not pure ASCII and contain any invalid UTF-8 characters are processed using the system default code page. No attempt is made to detect whether a different legacy code page is more likely to be correct, or is explicitly declared within the file.
The sad fact about file encoding is that in most cases there is no algorithmic way to be absolutely certain about what it is. (The exceptions are pure ASCII — but then, if any non-ASCII characters are introduced, there is no way to know how those should be encoded — and some file types, like HTML and XML, that can include a declaration of their own encoding, the declaration itself being in plain ASCII. Byte order marks are, in practice, another exception: they could occur in a legacy encoding, but they would be so uncommon at the start of a file that it’s safe to assume they are definitive, or the file has been purposely engineered to cause mis-detection.)
I will probably improve handling of file encodings as Search++ evolves. I doubt that I will attempt to duplicate exactly what Notepad++ does.
