@guy038 said:
Sorry, @coises, but I came acroos some differences of counting, in regex mode, between your two plugins Columns++ and Search++
It will take me some time to test, but my guess is that the differences come from this:
Search++ uses the properties returned by the ICU 78.3 (Unicode 17) API directly.
Columns++ uses a Python script to condense information from the Unicode Character Database tables into static C++ structures which can be searched quickly at run time. Subsequent tests (I think it was you who first ran them) showed that my method does not capture all relevant Unicode properties for all characters correctly.
Notepad++ and MultiReplace use whatever Boost.regex does by default when matching UTF-16 or a legacy codepage. I think this relies entirely on Windows character classification routines.
I expect that the results in Search++ using Regex and ICU will be identical. Any discrepancies would probably indicate bugs in the Regex implementation; the possibility of running tests like that is why I included ICU in the first place.
Edit to add:
If you simply count word characters (\w) in your Total_Chars.txt file, you’ll see that Notepad++, Columns++ and Search++ (Regex or ICU) come up with different counts:
Notepad++ native: 48,031
Columns++: 146,443
Search++: 149,366
Regex in Search++ uses the definition in Unicode Technical Standard #18: Unicode Regular Expressions. ICU in Search++ uses the ICU4C implementation of regular expressions directly.
Columns++ defines a word character as General Categories Ll, Lm, Lo, Lt, Lu and Nd plus the underscore. Relative to the Unicode standard, it misses connector punctuation (except the underscore), digits that are not decimal digits, marks, characters that are classed as alphabetic but are not letters, and the join control characters.
Notepad++ native search and MultiReplace use Boost.Regex without customization for character traits, which relies on GetStringTypeExW. I think a word character is defined as one which returns the flag for C1_ALPHA or C1_DIGIT or is an underscore.