Hi All,
Continuation of the previous script :
Then , with the help of the excellent Babel Map software, updated for Unicode v13.0
https://www.babelstone.co.uk/Software/BabelMap.html
I succeeded to create a list of the 21,143 remaining characters, from the living scripts, above, which should be truly considered as word character, without any ambiguity
However, when applying the regex \t\w\t against this list, I got a total of 17,307 word characters, only, because, probably, Notepad++ does not use the Boost regex library with full Unicode support :
The Boost definition of the regex \w does not consider all the characters over the BMP
Some characters of the BMP, although alphabetic, are not considered, yet, as word chars
For instance, in this short list, below, each Unicode char, surrounded with two tabulation chars, cannot be found with the regex \t\w\t, although it is, indeed, seen as a word by the Unicode Consortium` :-((
023D Ƚ ; Upper_Letter # Lu LATIN CAPITAL LETTER L WITH BAR
0370 Ͱ ; Upper_Letter # Lu GREEK CAPITAL LETTER HETA
04CF ӏ ; Lower_Letter # Ll CYRILLIC SMALL LETTER PALOCHKA
066F ٯ ; Other_Letter # Lo ARABIC LETTER DOTLESS QAF
0D60 ൠ ; Other_Letter # Lo MALAYALAM LETTER VOCALIC RR
200D ; Join_Control # Cf ZERO WIDTH JOINER
213F ℿ ; Upper_Letter # Lu DOUBLE-STRUCK CAPITAL PI
2187 ↇ ; Letter_Numb. # Nl ROMAN NUMERAL FIFTY THOUSAND
24B6 Ⓐ ; Other_Alpha. # So CIRCLED LATIN CAPITAL LETTER A
2E2F ⸯ ; Modifier_Let # Lm VERTICAL TILDE
A727 ꜧ ; Lower_Letter # Ll LATIN SMALL LETTER HENG
FF3F ＿ ; Conn._Punct. # Pc FULLWIDTH LOW LINE
1D400 𝐀 ; Upper_Letter # Lu MATHEMATICAL BOLD CAPITAL A
1D70B 𝜋 ; Lower_Letter # Ll MATHEMATICAL ITALIC SMALL PI
1F150 🅐 ; Other_Alpha. # So NEGATIVE CIRCLED LATIN CAPITAL LETTER A
To my mind, for all these reasons, as we cannot rely on the Word notion, the View > Summary... feature should just ignore the number of words or, at least, add the indication With caution !
By contrast, I think that it would be useful to count the number of Non_Space strings, determined with the regex \S+. Indeed, we would get more confident results ! The boundaries of Non_Space strings, which are the Space characters, belong to the well-defined list of the 25 Unicode characters with the binary property White_Space, from the PropList.txt file. Refer to the very beginning of this file :
http://www.unicode.org/Public/UCD/latest/ucd/PropList.txt
As a reminder, the regex \s, identical to \h|\v. So, it represents the complete character class [\t\x20\xA0\x{1680}\x{2000}-\x{200A}\x{202F}\x{205F}\x{3000}]|[\n\x0B\f\r\x85\x{2028}\x{2029}] which can be re-ordered as :
\s = [\t\n\x0B\f\r\x20\x85\xA0\x{1680}\x{2000}-\x{200A}\x{202F}\x{2028}\x{2029}\x{205F}\x{3000}]
Note that, in practice, the \s regex is mainly equivalent to the simple regex [\t\n\r\x20]
Here is that Unicode list of all Unicode characters with the property White_Space, with their name and their General_Category value :
0009 TAB ; White_Space # Cc TABULATION <control-0009>
000A LF ; White_Space # Cc LINE FEED <control-000A>
000B ; White_Space # Cc VERTICAL TABULATION <control-000B>
000C ; White_Space # Cc FORM FEED <control-000C>
000D CR ; White_Space # Cc CARRIAGE RETURN <control-000D>
0020 ; White_Space # Zs SPACE
0085
; White_Space # Cc NEXT LINE <control-0085>
00A0 ; White_Space # Zs NO-BREAK SPACE
1680 ; White_Space # Zs OGHAM SPACE MARK
2000 ; White_Space # Zs EN QUAD
2001 ; White_Space # Zs EM QUAD
2002 ; White_Space # Zs EN SPACE
2003 ; White_Space # Zs EM SPACE
2004 ; White_Space # Zs THREE-PER-EM SPACE
2005 ; White_Space # Zs FOUR-PER-EM SPACE
2006 ; White_Space # Zs SIX-PER-EM SPACE
2007 ; White_Space # Zs FIGURE SPACE
2008 ; White_Space # Zs PUNCTUATION SPACE
2009 ; White_Space # Zs THIN SPACE
200A ; White_Space # Zs HAIR SPACE
2028
; White_Space # Zl LINE SEPARATOR
2029
; White_Space # Zp PARAGRAPH SEPARATOR
202F ; White_Space # Zs NARROW NO-BREAK SPACE
205F ; White_Space # Zs MEDIUM MATHEMATICAL SPACE
3000 ; White_Space # Zs IDEOGRAPHIC SPACE
Note that I used the notations TAB, LF and CR, standing for the three characters \t, \n and \r, instead of the chars themselves
So, in order to get the number of Non_Space stings, we should use the simple regex \S+. However, it does not give the right number. Indeed, when several characters, with code-point over the BMP, are consecutive, they are not seen as a global Non_Space string but as individual ones :-((
Test my statement with this string, composed of four consecutive emoji chars 👨👩👦👧. The regex \S+ returns four Non_Space chars, whereas I would have expected only one string !
Consequently, I verified that the suitable regex to count all the Non_Space strings of a file, whatever their Unicode code-point, is rather the regex (?s)((?!\s).[\x{D800}-\x{DFFF}]?)+ ( Longer, I agree but exact ! )
Now, here is a list of regexes which allow you to count different subsets of characters from the current file contents :
- In a current non-ANSI ( UNICODE ) document, as the zone [\x{D800}-\x{DFFF}] represents the reserved SURROGATE area :
- Number of chars, in range [U+0000 - U+007F ], WITHOUT the \r AND \n chars = N1 = (?![\r\n])[\x{0000}-\x{007F}]
- Number of chars, in range [U+0080 - U+07FF ] = N2 = [\x{0080}-\x{07FF}]
- Number of chars, in range [U+0800 - U+FFFF ], except in SURROGATE range [D800 - DFFF] = N3 = (?![\x{D800}-\x{DFFF}])[\x{0800}-\x{FFFF}]
------------------------------------------------
- Number of chars, in range [U+0000 - U+FFFF ], in BMP , WITHOUT the \r AND \n = N1 + N2 + N3 = (?![\r\n\x{D800}-\x{DFFF}])[\x{0000}-\x{FFFF}]
- Number of chars, in range [U+10000 - U+10FFFF], OVER the BMP = N4 = (?-s).[\x{D800}-\x{DFFF}]
------------------------------
- TOTAL chars, in a NON-ANSI document, WITHOUT the \r AND \r chars = N1 + N2 + N3 + N4 = (?s)(?![\r\n]). or [^\r\n]
- Number of \r characters + Number of \n characters = N0 = \r|\n
------------------------------
- TOTAL chars, in a NON-ANSI document, WITH the \r AND \r chars = N0 + N1 + N2 + N3 + N4 = (?s). or (?s).[\x{D800}-\x{DFFF}]?
- In a current ANSI document :
- Number of characters, in range [U+0000 - U+00FF], WITHOUT the \r AND \n chars = N1 = (?s)(?![\r\n]). or [^\r\n]
- Number of \r characters + Number of \n characters = N0 = \r|\n
------------------------------
- TOTAL chars, in an ANSI document, WITH the \r AND \r chars = N0 + N1 = (?s).
- TOTAL current DOCUMENT length <Lb> in Notepad++ BUFFER :
- In an ANSI document Lb = N0 + N1 = (?s).
- In an UTF-8 or UTF-8-BOM document Lb = N0 + N1 + 2 × N2 + 3 × N3 + 4 × N4
- In an UCS-2 BE BOM or UCS-2 BE BOM document Lb = ( N0 + N1 + N2 + N3 ) × 2 = (?s). × 2
- Byte Order Mark ( BOM = U+FEFF ) length <Bl> and encoding, for SAVED documents :
- In an ANSI or UTF-8 document Bl = 0 byte
- In an UTF-8-BOM document Bl = 3 bytes ( EF BB BF )
- In an UCS-2 BE BOM document Bl = 2 bytes ( FE FF )
- In an UCS-2 LE BOM document Bl = 2 bytes ( FF FE )
- TOTAL current FILE length <Ld> on DISK for SAVED documents :
- WHATEVER the encoding of the document Ld = Lb + Bl ( = Total DOCUMENT length + BOM length )
- NUMBER of WORDS = \w+ ( to be considered with CAUTION )
- NUMBER of NON_SPACE strings = (?s)((?!\s).[\x{D800}-\x{DFFF}]?)+
- Number of LINES :
- Number of true EMPTY lines = (?<![\f\x{0085}\x{2028}\x{2029}])^(?:\r\n|\r|\n)
- Number of lines containing TAB and/or SPACE characters ONLY = (?<![\f\x{0085}\x{2028}\x{2029}])^[\t\x20]+(?:\r\n|\r|\n|\z)
-----------------------------------------------------------
- Number of BLANK or EMPTY lines = (?<![\f\x{0085}\x{2028}\x{2029}])^[\t\x20]*(?:\r\n|\r|\n|\z)
- Number of NON BLANK and NON EMPTY lines = (?!^[\t\x20]+$)(?:.|[\f\x{0085}\x{2028}\x{2029}])+(?:\r\n|\r|\n|\z)
---------------------------------------------------------------------
- TOTAL number of LINES = \r\n|\r|\n|(?:.|[\f\x{0085}\x{2028}\x{2029}])\z
Remarks : Although most of the regexes, above, can be easily understood, here are some additional elements :
The regex (?-s).[\x{D800}-\x{DFFF}] is the sole correct syntax, with our Boost regex engine, to count all the characters over the BMP
The regex (?s)((?!\s).[\x{D800}-\x{DFFF}]?)+, to count all the Non_Space strings, was explained before
In all the regexes, relative to the counting of lines, you probably noticed the character class [\f\x{0085}\x{2028}\x{2029}]. It must be present because the four characters \f, \x{0085} , \x{2028} and \x{2029} are, both, considered as a start and an End of line, like the assertions ^ and $ !
For instance, a single line, containing only one Next_Line char ( NEL ), of code-point \x{0085}, is wrongly seen as an empty line by the simple regex ^(?:\r\n|\r|\n) which matches the line-break after the Next_Line char !
To end , I would like to propose a new layout of an summary feature, which should be more informative ! Of course, the remaining of each line, after the colon, is only regexes, comments or descriptive areas !
Full File Path : X:\....\....\
Creation Date : MM/JJ/YYYY HH:MM:SS UTF-8[-BOM] UCS-2 BE/LE BOM ANSI
Modification Date : MM/JJ/YYYY HH:MM:SS --------------------------------------------------------------------------------------
1-Byte Chars : N1 = (?![\r\n])[\x{0000}-\x{007F}] idem [^\r\n]
2-Bytes Chars : N2 = [\x{0080}-\x{07FF}] idem 0
3-Bytes Chars : N3 = (?![\x{D800}-\x{DFFF}])[\x{0800}-\x{FFFF}] idem 0
Total BMP Chars : N1 + N2 + N3 = (?![\r\n\x{D800}-\x{DFFF}])[\x{0000}-\x{FFFF}] idem [^\r\n]
4-Bytes Chars : N4 = (?-s).[\x{D800}-\x{DFFF}] 0 0
Chars w/o CR|LF : N1 + N2 + N3 + N4 = [^\r\n]
EOL ( CR or LF ) : N0 = \r|\n
TOTAL Characters : N0 + N1 + N2 + N3 + N4 = (?s).
N++ BUFFER Length : = N0 + N1 + 2 × N2 + 3 × N3 + 4 × N4 ( N0 + N1 + N2 + N3 ) × 2 (?s).
Byte Order Mark : = 0 ( UTF-8) or 3 ( UTF-8-BOM ) 2 0
FILE Length : BUFFER length + BOM
WORDS ( Caution ) : \w+
NON-SPACE strings : (?s)((?!\s).[\x{D800}-\x{DFFF}]?)+
True EMPTY lines : (?<![\f\x{0085}\x{2028}\x{2029}])^(?:\r\n|\r|\n)
BLANK lines : (?<![\f\x{0085}\x{2028}\x{2029}])^[\t\x20]+(?:\r\n|\r|\n|\z)
EMPTY/BLANK lines : (?<![\f\x{0085}\x{2028}\x{2029}])^[\t\x20]*(?:\r\n|\r|\n|\z)
NON-BLANK lines : (?!^[\t\x20]+$)(?:.|[\f\x{0085}\x{2028}\x{2029}])+(?:\r\n|\r|\n|\z)
TOTAL lines : \r\n|\r|\n|(?:.|[\f\x{0085}\x{2028}\x{2029}])\z
Selection(s) : X characters (Y bytes) in Z ranges
Best Regards,
guy038
