Columns++: Where regex meets Unicode (Here there be dragons!)
-
Hi, @coises,
Two points :
-
Seemingly, if I select all the text of a regex, it does not appear automatically in the
Find What :zone and I need aCtrl + C/Ctrl + Voperation. Is this on purpose ? -
You may enter a very long line of text in the
Find What :zone. I verified that you can add up to30,000chars if it does not contain any line-break. So my question is :
Is it possible to enter a multi-lines text in the search zone of
ColumnsPlusPlus?TIA,
BR
guy038
-
-
@guy038 said in Columns++: Where regex meets Unicode (Here there be dragons!):
With your second experimental version of Columns++,
114collating elements can be found. Whaouh !There are actually 116 in that group, since
[.LF.]and[.CR.]are also included.However, the following FOUR ones cannot be reached although they are format characters ( NOT important )
| 1BCA0 | SHORTHAND FORMAT LETTER OVERLAP | [.SFLO.] | | 1BCA1 | SHORTHAND FORMAT CONTINUING OVERLAP | [.SFCO.] | | 1BCA2 | SHORTHAND FORMAT DOWN STEP | [.SFDS.] | | 1BCA3 | SHORTHAND FORMAT UP STEP | [.SFUS.] |I’ll add those.
BTW, there are
31di-graph characters, which are, either, considered as upper case and lower case letters, and which can be found with the Unicode class char\p{Lt}. With our present Boost regex engine, it correctly adds it, both, as an upper and lower letter !I see that indeed, when using Notepad++,
(?-i)\u(?<=\l)matches 31 characters. I’m not yet convinced that is desirable, though. Shouldn’t\l/\u,[:lower:]/[:upper:]and[:Ll:]/[:Lu:]all be the same? The title case characters are[:Lt:].However, an odd thing is the result of the
[[:cntrl:]]characters class : normally, as I said above, it should be65, so\p{Cc}(32for the CO controls, +DEL+32for the C1 control codes!It wasn’t clear to me how the POSIX
[:cntrl:]definition should be applied to Unicode. Notepad++ search appears to include most of the Cc and Cf characters in the basic multilingual plane, so I made it Cc + Cf. I’ll change that to Cc only.Thank you so much for looking so closely at all of this!
-
This post is deleted! -
@guy038 said in Columns++: Where regex meets Unicode (Here there be dragons!):
- Seemingly, if I select all the text of a regex, it does not appear automatically in the
Find What :zone and I need aCtrl + C/Ctrl + Voperation. Is this on purpose ?
It is. My original motivation for including a search function in Columns++ was to make it possible to search in rectangular selections — something Notepad++ search will not do. Thus, I expected that the most common way of using it would be to select the rectangular block in which you want to search, then open the dialog. (In a rectangular selection,
^and$match the beginning and end of the selection in each row. I attempt to explain the whole thing here.)After some feedback, I made it so that the initial selection is used to set an indicator to the region to be searched. That made sequential finds make a lot more sense.
My search has been subject to “mission creep” as I added formulas in replacement text, the Select options on the Count drop-down, the ability to convert multiple selections to search regions, and now 32-bit Unicode searching. Some day I might make a separate search plugin (or try to make a case for adding these features to Notepad++); for now, the one in Columns++ will be first oriented toward working conveniently with rectangular selections.
- You may enter a very long line of text in the
Find What :zone. I verified that you can add up to30,000chars if it does not contain any line-break. So my question is :
Is it possible to enter a multi-lines text in the search zone of
ColumnsPlusPlus?At present, no. It’s a good idea, though. I’ve thought of making it possible to open a separate window in which to enter search and replacement expressions, allowing more space and maybe containing a feature to pin frequently-used searches and/or a “builder” that would guide novices in the construction of regular expressions.
That’s getting so complex, though, that I think it might have to wait for that apocryphal day when I build a separate plugin that’s just for search.
- Seemingly, if I select all the text of a regex, it does not appear automatically in the
-
Hello, @coises and All,
Thanks for adding the
4remaining elements : so we’ll get a round number of collating elements :120!You said :
Notepad++ search appears to include most of the Cc and Cf characters in the basic multilingual plane, so I made it Cc + Cf. I’ll change that to Cc only
I confirm that, in your second version,
[[:cntrl:]]=\p{Cc}+\p{Cf}= 65 + 170 =235and thanks for the future modification
You said :
Shouldn’t
\l/\u,[:lower:]/[:upper:]and[:Ll:]/[:Lu:]all be the same? The title case characters are[:Lt:].What do you want to say ? Presently, in your second version, it’s just the case, as shown below or may I miss something obvious !?
[[:upper:]] = \p{upper} = [[:u:]] = \p{u} = \pu = \u = \p{Lu} = \p{Uppercase Letter} = [[:Lu:]] an UPPER case letter = 1,858 [[:lower:]] = \p{lower} = [[:l:]] = \p{l} = \pl = \l = \p{Ll} = \p{Lowercase Letter} = [[:Ll:]] a LOWER case letter = 2,258
BTW, I didn’t know that the syntax of an Unicode character class
\p{Xy}could also be expressed as[[:Xy:]]!Best Regards,
guy038
-
@guy038 said in Columns++: Where regex meets Unicode (Here there be dragons!):
You said :
Shouldn’t
\l/\u,[:lower:]/[:upper:]and[:Ll:]/[:Lu:]all be the same? The title case characters are[:Lt:].What do you want to say ? Presently, in your second version, it’s just the case, as shown below or may I miss something obvious !?
I don’t think you missed anything. I think I might have misunderstood you. I thought you were saying that
[:lower:]and[:upper:]and/or\land\ushould match the[:Lt:]characters, so that those 31 characters are both upper case and lower case. Perhaps we are agreed that they are neither.BTW, I didn’t know that the syntax of an Unicode character class
\p{Xy}could also be expressed as[[:Xy:]]!Boost::regex is built such that
\p{whatever}and[[:whatever:]]are the same. It also “delegates” backslash lower case letter escapes that don’t have any other meaning to classes with the same name, and upper case escapes without another meaning to the complements; so\sis internally “defined” as[[:s:]]and\Sas +[^[:s:]]. That’s how I was able to define\i,\m,\oand\y. It’s also why we have to write\p{L*}instead of\p{L}: class names are case insensitive, and “l” already defines\las lower case. For consistency, all the Unicode general category groups use the asterisk notation. -
I’m hopeful that the real end goal for all of this is integration into native Notepad++, and that the plugin is really just a “testbed” for what you’re doing. Columns++ is great, but this is about core unicode searching, and as such really belongs in the standard product.
It’s great that a person has finally been found that’s capable of (and interested in) doing this stuff, and it would be a shame if Notepad++ moves forward without the benefits of this work.
Thank you for your work.
-
@Alan-Kilborn said in Columns++: Where regex meets Unicode (Here there be dragons!):
I’m hopeful that the real end goal for all of this is integration into native Notepad++, and that the plugin is really just a “testbed” for what you’re doing. Columns++ is great, but this is about core unicode searching, and as such really belongs in the standard product.
For now I’m focusing on making the search in Columns++ as good as I can make it within the bounds of what I’ve intended search in Columns++ to accomplish. I don’t know that I can get this to where I’m comfortable calling it “stable” before the plugins list for the just-announced Notepad++ 8.7.8 release is frozen, but that’s the limit of my ambition at this point.
I do hope that once it has been in use for a time, it can serve as a proof of viability — and maybe a bit of pressure — to incorporate better Unicode searching into Notepad++. That would be a massive code change, though, and unfortunately not everything can be simply copied from the way I’m doing it. (Columns++ uses Boost::regex directly; Notepad++ integrates Boost::regex with Scintilla and then uses the upgraded Scintilla search. Most of the same principles should apply, but details, details… details are where the bugs live.)
There will also surely be a repeat of the same question I faced: whether to use ad hoc code or somehow incorporate ICU, which Boost::regex can use. And since Windows 10 version 1703 (but changing in 1709 and again in 1903), Windows incorporates a stripped-down version of ICU. It appears that Boost::regex can’t use that, but perhaps Boost will fix that someday, or perhaps I or someone else will find a way to connect them. By the time this could be considered for Notepad++, it might be plausible to limit new versions to Win 10 version 1903 or later. Avoiding bespoke code would minimize the possibility of future maintenance burdens for Notepad++. So there will be a lot to consider.
Thank you for your kind words and encouragement, Alan.
-
I’ve posted Columns++ for Notepad++ version 1.1.5.3-Experimental.
Changes:
-
Search in Columns++ shows a progress dialog when it estimates that a count, select or replace all operation will take more than two seconds. That should make apparent freezes (which were observed when attempting select all for expressions that make tens or hundreds of thousands of separate matches) far less likely to happen. (Note that this is not connected to the “Expression too complex” situation; this happens when the expression is reasonable, but there are an extremely high number of matches.)
-
[[:cntrl:]]matches only Unicode General Category Cc characters. Mnemonics for formatting characters[[.sflo.]],[[.sfco.]],[[.sfds.]]and[[.sfus.]]work. -
I corrected an error that would have caused equivalence classes (e.g.,
[[=a=]]) to fail for characters U+10000 and above. However, I don’t know if there are any working equivalence classes for characters U+10000 and above, anyway. (Present support for those is dependent on a Windows function; it appears to me that it might not process surrogate pairs in a useful way.) -
There were other organizational changes.
As always comments, observations and suggestions are most welcome. My aim is for this to be the last “experimental” release in this series, if nothing awful happens… in which case the major remaining thing to be done before a normal release is documentation.
-
-
Hi, @coises and All,
First, here is the summary of the contents of the
Total_Chars.txtfile :•----------------•---------•------------------------------------------•-----------•-------------------------------------------------•-----------• | Range | Plane | COUNT / MARK of ALL characters | # Chars | COUNT / MARK of ALL UNASSIGNED characters | # Unas. | •----------------•---------•------------------------------------------•-----------•-------------------------------------------------•-----------• | 0000...FFFD | 0 | [\x{0000}-\x{FFFD}] | 63,454 | (?=[\x{0000}-\x{D7FF}]|[\x{F900}-\x{FFFD}])\Y | 1,398 | | 10000..1FFFD | 1 | [\x{10000}-\x{1FFFD}] | 65,534 | (?=[\x{10000}-\x{1FFFD}])\Y | 37,090 | | 20000..2FFFD | 2 | [\x{20000}-\x{2FFFD}] | 65,534 | (?=[\x{20000}-\x{2FFFD}])\Y | 4,039 | | 30000..3FFFD | 3 | [\x{30000}-\x{3FFFD}] | 65,534 | (?=[\x{30000}-\x{3FFFD}])\Y | 56,403 | | E0000..EFFFD | 14 | [\x{E01F0}-\x{EFFFD}] | 65,534 | (?=[\x{E0000}-\x{EFFFD}])\Y | 65,197 | •----------------•---------•------------------------------------------•-----------•-------------------------------------------------•-----------• | 00000..EFFFD | | (?s). \I \p{Any} [\x0-\x{EFFFD}] | 325,590 | (?![\x{E000}-\x{F8FF}])\Y \p{Not Assigned} | 164,127 | •----------------•---------•------------------------------------------•-----------•-------------------------------------------------•-----------•Indeed, I cannot post my new
Unicode_Col++.txtfile, in its entirety, with the detail of all the Unicode blocks ( Too large ! ). However, it will be part of my futureUnicode.ziparchive that I’ll post on myGoogle Driveaccount !
Now, I tested your third experimental version of
Columns++and everything works as you surely expect to !!You said :
Search in Columns++ shows a progress dialog when it estimates that a count, select or replace all operation will take more than two seconds…
I pleased to tell you that with this new feature, my laptop did not hang on any more ! For example, I tried to select all the matches of the regex
(?s)., against myTotal_Chars.txtfile and, with the process dialog on my HP ProBook 450 G8 / Windows 10 Pro 64 / Version 21H1 / Intel® Core™ i7 / RAM 32 GB DDR4-3200 MHz, after8 m 21s, the green zone was complete and it said :325590 matches selected! I even copied all this selection on a new tab and, after suppression of all\r\nline-breaks, theComparePlusplugin did not find any difference betweenTotal_Chars.txtand this new tab !
You said :
[[:cntrl:]]matches only Unicode General Category Cc characters. Mnemonics for formatting characters[[.sflo.]],[[.sfco.]],[[.sfds.]]and[[.sfus.]]work.I confirm that these two changes are effective
Now, I particularly tested the
Equivalenceclasses feature. You can refer to the following link :https://unicode.org/charts/collation/index.html
And also consult the help at :
https://unicode.org/charts/collation/help.html
For the letter
a, it detects160equivalences of aaletterHowever, against the
Total_Chars.txtfile, the regex[[=a=]]returns86matches. So we can deduce that :-
A lot of equivalences are not found with the
[[=a=]]regex -
Some equivalents, not shown from this link, can be found with the
[[=a=]]regex. it’s the case with the\x{249C}character ( PARENTHESIZED LATIN SMALL LETTER A ) !
This situation happens with any character : for example, the regex
[[=1=]]finds54matches, but, on the site, it shows209equivalences to the digit1Now, with your experimental
UTF-32version, you can use any other equivalent character of thealetter to get the86matches (((=Ⱥ=]],[[=ⱥ=]],[[=Ɐ=]], … ). Note that, with our presentBoostregex engine, some equivalences do not return the86matches. It’s the case for the regexes :[[=ɐ=]],[[=ɑ=]],[[=ɒ=]],[[=ͣ=]],[[=ᵃ=]],[[=ᵄ=]],[[=ⱥ=]],[[=Ɑ=]],[[=Ɐ=]],[[=Ɒ=]]Thus, your version is more coherent, as it does give the same result, whatever the char used in the equivalence class regex !
Here is below the list of all the equivalences of any char of the
Windows-1252code-page, from\x{0020}till\x{00DE}Note that, except for the DEL character, as en example, I did not consider the equivalence classes which return only one match !I also confirm, that I did not find any character over
\x{FFFF}which would be part of a regex equivalence class, either with our Boost engine or with yourColumns++experimental version ![[= =]] = [[=space=]] => 3 ( ) [[=!=]] = [[=exclamation-mark=]] => 2 ( !！ ) [[="=]] = [[=quotation-mark=]] => 3 ( "⁍＂ ) [[=#=]] = [[=number-sign=]] => 4 ( #؞⁗＃ ) [[=$=]] = [[=dollar-sign=]] => 3 ( $⁒＄ ) [[=%=]] = [[=percent-sign=]] => 3 ( %⁏％ ) [[=&=]] = [[=ampersand=]] => 3 ( &⁋＆ ) [[='=]] = [[=apostrophe=]] => 2 ( '＇ ) [[=(=]] = [[=left-parenthesis=]] => 4 ( (⁽₍（ ) [[=)=]] = [[=right-parenthesis=]] => 4 ( )⁾₎） ) [[=*=]] = [[=asterisk=]] => 2 ( *＊ ) [[=+=]] = [[=plus-sign=]] => 6 ( +⁺₊﬩﹢＋ ) [[=,=]] = [[=comma=]] => 2 ( ,， ) [[=-=]] = [[=hyphen=]] => 3 ( -﹣－ ) [[=.=]] = [[=period=]] => 3 ( .․． ) [[=/=]] = [[=slash=]] => 2 ( /／ ) [[=0=]] = [[=zero=]] => 48 ( 0٠۟۠۰߀०০੦૦୦୵௦౦౸೦൦๐໐༠၀႐០᠐᥆᧐᪀᪐᭐᮰᱀᱐⁰₀↉⓪⓿〇㍘꘠ꛯ꠳꣐꤀꧐꩐꯰０ ) [[=1=]] = [[=one=]] => 54 ( 1¹١۱߁१১੧૧୧௧౧౹౼೧൧๑໑༡၁႑፩១᠑᥇᧑᧚᪁᪑᭑᮱᱁᱑₁①⑴⒈⓵⚀❶➀➊〡㋀㍙㏠꘡ꛦ꣑꤁꧑꩑꯱１ ) [[=2=]] = [[=two=]] => 54 ( 2²ƻ٢۲߂२২੨૨୨௨౨౺౽೨൨๒໒༢၂႒፪២᠒᥈᧒᪂᪒᭒᮲᱂᱒₂②⑵⒉⓶⚁❷➁➋〢㋁㍚㏡꘢ꛧ꣒꤂꧒꩒꯲２ ) [[=3=]] = [[=three=]] => 53 ( 3³٣۳߃३৩੩૩୩௩౩౻౾೩൩๓໓༣၃႓፫៣᠓᥉᧓᪃᪓᭓᮳᱃᱓₃③⑶⒊⓷⚂❸➂➌〣㋂㍛㏢꘣ꛨ꣓꤃꧓꩓꯳３ ) [[=4=]] = [[=four=]] => 51 ( 4٤۴߄४৪੪૪୪௪౪೪൪๔໔༤၄႔፬៤᠔᥊᧔᪄᪔᭔᮴᱄᱔⁴₄④⑷⒋⓸⚃❹➃➍〤㋃㍜㏣꘤ꛩ꣔꤄꧔꩔꯴４ ) [[=5=]] = [[=five=]] => 53 ( 5Ƽƽ٥۵߅५৫੫૫୫௫౫೫൫๕໕༥၅႕፭៥᠕᥋᧕᪅᪕᭕᮵᱅᱕⁵₅⑤⑸⒌⓹⚄❺➄➎〥㋄㍝㏤꘥ꛪ꣕꤅꧕꩕꯵５ ) [[=6=]] = [[=six=]] => 52 ( 6٦۶߆६৬੬૬୬௬౬೬൬๖໖༦၆႖፮៦᠖᥌᧖᪆᪖᭖᮶᱆᱖⁶₆ↅ⑥⑹⒍⓺⚅❻➅➏〦㋅㍞㏥꘦ꛫ꣖꤆꧖꩖꯶６ ) [[=7=]] = [[=seven=]] => 50 ( 7٧۷߇७৭੭૭୭௭౭೭൭๗໗༧၇႗፯៧᠗᥍᧗᪇᪗᭗᮷᱇᱗⁷₇⑦⑺⒎⓻❼➆➐〧㋆㍟㏦꘧ꛬ꣗꤇꧗꩗꯷７ ) [[=8=]] = [[=eight=]] => 50 ( 8٨۸߈८৮੮૮୮௮౮೮൮๘໘༨၈႘፰៨᠘᥎᧘᪈᪘᭘᮸᱈᱘⁸₈⑧⑻⒏⓼❽➇➑〨㋇㍠㏧꘨ꛭ꣘꤈꧘꩘꯸８ ) [[=9=]] = [[=nine=]] => 50 ( 9٩۹߉९৯੯૯୯௯౯೯൯๙໙༩၉႙፱៩᠙᥏᧙᪉᪙᭙᮹᱉᱙⁹₉⑨⑼⒐⓽❾➈➒〩㋈㍡㏨꘩ꛮ꣙꤉꧙꩙꯹９ ) [[=:=]] = [[=colon=]] => 2 ( :： ) [[=;=]] = [[=semicolon=]] => 3 ( ;;； ) [[=<=]] = [[=less-than-sign=]] => 3 ( <﹤＜ ) [[===]] = [[=equals-sign=]] => 5 ( =⁼₌﹦＝ ) [[=>=]] = [[=greater-than-sign=]] => 3 ( >﹥＞ ) [[=?=]] = [[=question-mark=]] => 2 ( ?？ ) [[=@=]] = [[=commercial-at=]] => 2 ( @＠ ) [[=A=]] => 86 ( AaªÀÁÂÃÄÅàáâãäåĀāĂ㥹ǍǎǞǟǠǡǺǻȀȁȂȃȦȧȺɐɑɒͣᴀᴬᵃᵄᶏᶐᶛᷓḀḁẚẠạẢảẤấẦầẨẩẪẫẬậẮắẰằẲẳẴẵẶặₐÅ⒜ⒶⓐⱥⱭⱯⱰＡａ ) [[=B=]] => 29 ( BbƀƁƂƃƄƅɃɓʙᴃᴮᴯᵇᵬᶀḂḃḄḅḆḇℬ⒝ⒷⓑＢｂ ) [[=C=]] => 40 ( CcÇçĆćĈĉĊċČčƆƇƈȻȼɔɕʗͨᴄᴐᵓᶗᶜᶝᷗḈḉℂ℃ℭ⒞ⒸⓒꜾꜿＣｃ ) [[=D=]] => 44 ( DdÐðĎďĐđƊƋƌƍɗʤͩᴅᴆᴰᵈᵭᶁᶑᶞᷘᷙḊḋḌḍḎḏḐḑḒḓⅅⅆ⒟ⒹⓓꝹꝺＤｄ ) [[=E=]] => 82 ( EeÈÉÊËèéêëĒēĔĕĖėĘęĚěƎƏǝȄȅȆȇȨȩɆɇɘəɚͤᴇᴱᴲᵉᵊᶒᶕḔḕḖḗḘḙḚḛḜḝẸẹẺẻẼẽẾếỀềỂểỄễỆệₑₔ℮ℯℰ⅀ⅇ⒠ⒺⓔⱸⱻＥｅ ) [[=F=]] => 22 ( FfƑƒᵮᶂᶠḞḟ℉ℱℲⅎ⒡ⒻⓕꜰꝻꝼꟻＦｆ ) [[=G=]] => 45 ( GgĜĝĞğĠġĢģƓƔǤǥǦǧǴǵɠɡɢɣɤʛˠᴳᵍᵷᶃᶢᷚᷛḠḡℊ⅁⒢ⒼⓖꝾꝿꞠꞡＧｇ ) [[=H=]] => 41 ( HhĤĥĦħȞȟɥɦʜʰʱͪᴴᶣḢḣḤḥḦḧḨḩḪḫẖₕℋℌℍℎℏ⒣ⒽⓗⱧⱨꞍＨｈ ) [[=I=]] => 61 ( IiÌÍÎÏìíîïĨĩĪīĬĭĮįİıƖƗǏǐȈȉȊȋɨɩɪͥᴉᴵᵎᵢᵻᵼᶖᶤᶥᶦᶧḬḭḮḯỈỉỊịⁱℐℑⅈ⒤ⒾⓘꟾＩｉ ) [[=J=]] => 23 ( JjĴĵǰȷɈɉɟʄʝʲᴊᴶᶡᶨⅉ⒥ⒿⓙⱼＪｊ ) [[=K=]] => 38 ( KkĶķĸƘƙǨǩʞᴋᴷᵏᶄᷜḰḱḲḳḴḵₖK⒦ⓀⓚⱩⱪꝀꝁꝂꝃꝄꝅꞢꞣＫｋ ) [[=L=]] => 56 ( LlĹĺĻļĽľĿŀŁłƚƛȽɫɬɭɮʟˡᴌᴸᶅᶩᶪᶫᷝᷞḶḷḸḹḺḻḼḽₗℒℓ⅂⅃⒧ⓁⓛⱠⱡⱢꝆꝇꝈꝉꞀꞁＬｌ ) [[=M=]] => 33 ( MmƜɯɰɱͫᴍᴟᴹᵐᵚᵯᶆᶬᶭᷟḾḿṀṁṂṃₘℳ⒨ⓂⓜⱮꝳꟽＭｍ ) [[=N=]] => 47 ( NnÑñŃńŅņŇňʼnƝƞǸǹȠɲɳɴᴎᴺᴻᵰᶇᶮᶯᶰᷠᷡṄṅṆṇṈṉṊṋⁿₙℕ⒩ⓃⓝꞤꞥＮｎ ) [[=O=]] => 106 ( OoºÒÓÔÕÖØòóôõöøŌōŎŏŐőƟƠơƢƣǑǒǪǫǬǭǾǿȌȍȎȏȪȫȬȭȮȯȰȱɵɶɷͦᴏᴑᴒᴓᴕᴖᴗᴼᵒᵔᵕᶱṌṍṎṏṐṑṒṓỌọỎỏỐốỒồỔổỖỗỘộỚớỜờỞởỠỡỢợₒℴ⒪ⓄⓞⱺꝊꝋꝌꝍＯｏ ) [[=P=]] => 33 ( PpƤƥɸᴘᴾᵖᵱᵽᶈᶲṔṕṖṗₚ℘ℙ⒫ⓅⓟⱣⱷꝐꝑꝒꝓꝔꝕꟼＰｐ ) [[=Q=]] => 16 ( QqɊɋʠℚ℺⒬ⓆⓠꝖꝗꝘꝙＱｑ ) [[=R=]] => 64 ( RrŔŕŖŗŘřƦȐȑȒȓɌɍɹɺɻɼɽɾɿʀʁʳʴʵʶͬᴙᴚᴿᵣᵲᵳᶉᷢᷣṘṙṚṛṜṝṞṟℛℜℝ⒭ⓇⓡⱤⱹꝚꝛꝜꝝꝵꝶꞂꞃＲｒ ) [[=S=]] => 47 ( SsŚśŜŝŞşŠšƧƨƩƪȘșȿʂʃʅʆˢᵴᶊᶋᶘᶳᶴᷤṠṡṢṣṤṥṦṧṨṩₛ⒮ⓈⓢⱾꜱＳｓ ) [[=T=]] => 46 ( TtŢţŤťƫƬƭƮȚțȶȾʇʈʧʨͭᴛᵀᵗᵵᶵᶿṪṫṬṭṮṯṰṱẗₜ⒯ⓉⓣⱦꜨꜩꝷꞆꞇＴｔ ) [[=U=]] => 82 ( UuÙÚÛÜùúûüŨũŪūŬŭŮůŰűŲųƯưƱǓǔǕǖǗǘǙǚǛǜȔȕȖȗɄʉʊͧᴜᵁᵘᵤᵾᵿᶙᶶᶷᶸṲṳṴṵṶṷṸṹṺṻỤụỦủỨứỪừỬửỮữỰự⒰ⓊⓤＵｕ ) [[=V=]] => 29 ( VvƲɅʋʌͮᴠᵛᵥᶌᶹᶺṼṽṾṿỼỽ⒱ⓋⓥⱱⱴⱽꝞꝟＶｖ ) [[=W=]] => 28 ( WwŴŵƿǷʍʷᴡᵂẀẁẂẃẄẅẆẇẈẉẘ⒲ⓌⓦⱲⱳＷｗ ) [[=X=]] => 15 ( XxˣͯᶍẊẋẌẍₓ⒳ⓍⓧＸｘ ) [[=Y=]] => 36 ( YyÝýÿŶŷŸƳƴȲȳɎɏʎʏʸẎẏẙỲỳỴỵỶỷỸỹỾỿ⅄⒴ⓎⓨＹｙ ) [[=Z=]] => 41 ( ZzŹźŻżŽžƵƶȤȥɀʐʑᴢᵶᶎᶻᶼᶽᷦẐẑẒẓẔẕℤ℥ℨ⒵ⓏⓩⱫⱬⱿꝢꝣＺｚ ) [[=[=]] = [[=left-square-bracket=]] => 2 ( [［ ) [[=\=]] = [[=backslash=]] => 2 ( \＼ ) [[=]=]] = [[=right-square-bracket=]] => 2 ( ]］ ) [[=^=]] = [[=circumflex=]] => 3 ( ^ˆ＾ ) [[=_=]] = [[=underscore=]] => 2 ( _＿ ) [[=`=]] = [[=grave-accent=]] => 4 ( `ˋ`｀ ) [[={=]] = [[=left-curly-bracket=]] => 2 ( {｛ ) [[=|=]] = [[=vertical-line=]] => 2 ( |｜ ) [[=}=]] = [[=right-curly-bracket=]] => 2 ( }｝ ) [[=~=]] = [[=tilde=]] => 2 ( ~～ ) [[==]] = [[=DEL=]] => 1 ( ) [[=Œ=]] => 2 ( Œœ ) [[=¢=]] => 3 ( ¢《￠ ) [[=£=]] => 3 ( £︽￡ ) [[=¤=]] => 2 ( ¤》 ) [[=¥=]] => 3 ( ¥︾￥ ) [[=¦=]] => 2 ( ¦￤ ) [[=¬=]] => 2 ( ¬￢ ) [[=¯=]] => 2 ( ¯￣ ) [[=´=]] => 2 ( ´´ ) [[=·=]] => 2 ( ·· ) [[=¼=]] => 4 ( ¼୲൳꠰ ) [[=½=]] => 6 ( ½୳൴༪⳽꠱ ) [[=¾=]] => 4 ( ¾୴൵꠲ ) [[=Þ=]] => 6 ( ÞþꝤꝥꝦꝧ )
Some double-letter characters give some equivalences which allow you to get the right single char to use, instead of the two trivial letters :
[[=AE=]] = [[=Ae=]] = [[=ae=]] => 11 ( ÆæǢǣǼǽᴁᴂᴭᵆᷔ ) [[=CH=]] = [[=Ch=]] = [[=ch=]] => 0 ( ? ) [[=DZ=]] = [[=Dz=]] = [[=dz=]] => 6 ( DŽDždžDZDzdz ) [[=LJ=]] = [[=Lj=]] = [[=lj=]] => 3 ( LJLjlj ) [[=LL=]] = [[=Ll=]] = [[=ll=]] => 2 ( Ỻỻ ) [[=NJ=]] = [[=Nj=]] = [[=nj=]] => 3 ( NJNjnj ) [[=SS=]] = [[=Ss=]] = [[=ss=]] => 2 ( ßẞ )
However, the use of these di-graph characters are quite delicate ! Let’s consider these
7di-graph collating elements, below, with various cases :[[.AE.]] [[.Ae.]] [[.ae.]] ( European Ligature ) [[.CH.]] [[.Ch.]] [[.ch.]] ( Spanish ) [[.DZ.]] [[.Dz.]] [[.dz.]] ( Hungarian, Polish, Slovakian, Serbo-Croatian ) [[.LJ.]] [[.Lj.]] [[.lj.]] ( Serbo-Croatian ) [[.LL.]] [[.Ll.]] [[.ll.]] ( Spanish ) [[.NJ.]] [[.Nj.]] [[.nj.]] ( Serbo-Croatian ) [[.SS.]] [[.Ss.]] [[.ss.]] ( German )As we know that :
LJ 01C7 LATIN CAPITAL LETTER LJ Lj 01C8 LATIN CAPITAL LETTER L WITH SMALL LETTER J lj 01C9 LATIN SMALL LETTER LJ DZ 01F1 LATIN CAPITAL LETTER DZ Dz 01F2 LATIN CAPITAL LETTER D WITH SMALL LETTER Z dz 01F3 LATIN SMALL LETTER DZIf we apply the regex
[[.dz.]-[.lj.][=dz=][=lj=]]against the textbcddzʣefghiijjklljljmn, pasted in a new tab, Columns++ would find12matches :dz dz e f g h i j k l lj lj
To sum up, @coises, the key points, of your third experimental version, are :
-
A major regex engine, inplemented in UTF-32, which correctly handle all the Unicode characters, from
\x{0}to\x{0010FFFF}, and correctly manage all the Unicode character classes\p{Xy}or[[:Xy:]] -
Additional features as
\i,\m,\oand\yand their complements -
The
\Xregex feature (\M\m*) correctly works for characters OVER the BMP -
The invalid
UTF-8characters may be kept, replaced or deleted ( FIND\i+, REPLACEABC $1 XYZ) -
The
NULcharacter can be placed in replacement ( FINDABC\x00XYZ, REPLACE\x0--$0--\x{00}) -
Correct handle of case replacements, even in case of accentuated characters ( FIND
(?-s).REPLACE\U$0) -
The
\Kfeature ALSO works in a step-by-step replacement with theReplacebutton ( FIND^.{20}\K(.+), REPLACE--\1--)
To end, @coises, do you think it’s worth testing some regex examples with possible replacements ? I could test some tricky regexes to check the robustness of your final
UTF-32version., if necessary ?Best Regards,
guy038
-
-
@guy038 said:
The \K feature ALSO works in a step-by-step replacement with the Replace button
That’s major. Perhaps whatever change allows that could be factored out and put into native Notepad++?
(Again, I’m not one to say often that functionality that’s in a plugin should “go native”…but, when we’re talking about important find/replace functionality…it should).
I could test some tricky regexes to check the robustness of your final UTF-32 version
First, I’d encourage this further testing.
Second, is there a reason to mention UTF-32, specifically?
-
@guy038 said in Columns++: Where regex meets Unicode (Here there be dragons!):
To end, @coises, do you think it’s worth testing some regex examples with possible replacements ? I could test some tricky regexes to check the robustness of your final UTF-32 version., if necessary ?
It would surely be helpful; but I now know there will be at least one more experimental version, so you might as well wait for that. I no longer expect to have this ready in time to be included in the plugins list for the next Notepad++ release.
It turns out that
\Xdoes not work correctly. Consider this text:👍👍🏻👍🏼👍🏽👍🏾👍🏿There are six “graphical characters” there, but
\Xfinds eleven (if you copy without a line ending). It turns out the rules for identifying grapheme cluster breaks are complex, and Boost::regex does not implement them correctly. (As far as I can tell, Scintilla is agnostic about this. Selections go by code point — stepping with the right arrow key, you can see the cursor move to the middle of any character comprised of multiple code points. I think Scintilla depends on the fonts and the display engine to render grapheme clusters properly, but I haven’t verified that.)So I’m working on making that work properly. I think I’ve found a way, but work is still in progress.
I will also look at the equivalence classes problems you identified. Thank you for that information! It will help greatly.
I’ve had a couple thoughts, and I’m wondering what others think:
-
I find the character class matching when Match case is not checked (or
(?i)is used) absurd. Boost::regex makes\land\umatch all[[:alpha:]]characters (not just cased letters), and the Unicode classes become entirely erratic. I can’t think of any named character classes that would be less useful if case insensitivity were ignored when matching them. If it’s possible to do that — so that, for example,\ustill matches only upper case characters even when Match case is not checked — would others find that an improvement? Would anyone find it problematic? (This wouldn’t affect classes specified with explicit characters, like[aeiou]or[A-F]: Match case would still control how those match. If I can accomplish this as I intend, only the Unicode “General Category” character classes,\l,\u,[:lower:],[:upper:]and obvious correlates would be changed to ignore case insensitivity and always test the document text as written.) -
Should there be an option to make the POSIX classes and their escapes (such as
\s,\w,[[:alnum:]],[[:punct:]]) match only ASCII characters? Unfortunately, I don’t see any reasonable way to make that an in-expression switch like(?i); if it were done at all, it would have to be a checkbox that would apply to the entire expression. Would this help anyone, or just add complication for little value? -
Does anyone care much about having Unicode script properties available as regex properties (e.g.,
\p{Greek},\p{Hebrew},\p{Latin})? -
Does anyone care much about having Unicode character names available (e.g.,
[[.GREEK SMALL LETTER FINAL SIGMA.]]equivalent to\x{03C2})? My thought is that including those will make the module much larger, and that by the time you’ve looked up the exact way the name has to be given, you could just look up the hexadecimal code point anyway.
-
-
@Alan-Kilborn said in Columns++: Where regex meets Unicode (Here there be dragons!):
@guy038 said:
The \K feature ALSO works in a step-by-step replacement with the Replace button
That’s major. Perhaps whatever change allows that could be factored out and put into native Notepad++?
When doing a Replace, the first step is to check that the selection matches the search expression. In general, this fails when
\Kis used because an expression using\Kdoesn’t select starting from where it matched.What Columns++ does is to remember the starting position for the last successful find (whether it was from Find or as part of a Replace). Then, when Replace is clicked, it checks starting from there rather than from the selection.
It’s been a while since I wrote that code, and I don’t remember exactly why, but I put in a number of checks to be sure the remembered starting point is still valid. In particular, if focus leaves the Search dialog (meaning the user might have changed the selection or the text), the position memory is marked invalid and a normal search starting from the beginning of the selection is used. I think the reason was to be sure that if the user intends to start a new search (by clicking in a new position or changing the selection), it would be important not to start from some remembered (and now meaningless) point.
Off hand, I don’t see any reason the same principle couldn’t be applied to Notepad++ search. It wouldn’t be a matter of just copying, though; someone would have to think through the logic from scratch in the context of how Notepad++ implements search.
