Perl language syntax highlighting troubles (bug or limitation ?)

[Gilles Maisonneuve]

@Ekopalypse

Now, if I say in Pyhton (attempt to transliterate from Perl) :

(r'(?s)(\h*(<<)\h*["|']?([^"|^']+?)["|']?\h*;.*?\3)', [2])

does it mean :

1. form REGEXP
2. do not match NL with DOT
3. matches any horizontal blanks (0 or more), don’t make a group
4. matches ‘<<’ make it a group
5. matches any horizontal blanks (0 or more), don’t make a group
6. matches 0 or 1 text quote (either double or single), no group
7. matches a group of any chars not " nor ’ one or more time(s) (in perl it would be [^"'])
8. matches 0 or 1 text quote (either double or single), no group
9. possible blanks until semi-colon, semi-colon, then possible chars until NL

BUT THEN, what does mean ?\3. I’m lost there.

[Gilles Maisonneuve]

a slash m

[Ekopalypse]

the r at the beginning just informs python that this is a raw string and
every char must be taken literally otherwise backslashes would be treated
as escapes under some circumstances.

The regex string is only this part

(?s)(\h*(<<)\h*["|']?([^"|^']+?)["|']?\h*;.*?\3)

and I would say, but as said - not an regex expert at all,

(?s) means Dot matches newline characters
the first matching group is

(\h*(<<)\h*["|']?([^"|^']+?)["|']?\h*;.*?\3)

the second

(<<)

and the third must be

([^"|^']+?)

if I’m right.

\3 should be the same as $3 in perl

[Gilles Maisonneuve]

@Ekopalypse

still confused: ([^"|^']+?) why a ‘?’ after the ‘+’ what’s for this ‘?’

and then \3 would mean the 3rd matching group (third ‘()’) but in Perl is used only in subsitutions. What is the use here ? There are only 2 groups in the regex (two blocks surrounded by parenthèses only.

[Ekopalypse]

@Gilles-Maisonneuve

maybe this picture makes it a little bit clearer

[Ekopalypse]

@Gilles-Maisonneuve

still confused: ([^"|^']+?) why a ‘?’ after the ‘+’ what’s for this ‘?’

as less as possible - non-greedy

and then \3 would mean the 3rd matching group (third ‘()’) but in Perl is used only in >subsitutions. What is the use here ? There are only 2 groups in the regex (two blocks >surrounded by parenthèses only.

placeholder for what was found in match group 3, to find the EOT at the end

and there are 3 match groups or am I missing something??

[Gilles Maisonneuve]

@Ekopalypse

2 sets of parenteses only, where is the third set ?
so only 2 match groups

can you make this work :

no syntax error on the python console but absolutely no result, where is my bug ?

regexes[(3, (255,255,255))] = (r'(?s)(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)', [1])

[Ekopalypse]

@Gilles-Maisonneuve

[1] informs the python script, that only the results from sub match group 1 should be colored in white (255,255,255)
sub match group 1 is the result of (<<)

In order to make it painting all you can use [0]

I’m still confused about the 2 to 3 match groups.
Am I incorrect when saying that
(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)
(<<)
("{0,1}.+"{0,1})
are three match groups?

Maybe the confusion comes from the fact that references matches within a
regular expression starts by 1 but python starts counting match results by 0.

Sorry, but I have to stay up early tomorrow and it is already 1am but I’m really
interested in solving our (mis)understanding today later (maybe in ~16-18hours)?

[Gilles Maisonneuve]

ok, tomorrow is another day
‘see’ you tomorrow.
have a good night.
g

[Ekopalypse]

you too - see you

[Gilles Maisonneuve]

@Ekopalypse

OK, so the

(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)

is a regex group, not a function call surrounded by parenthèses or a logical group provided by the ‘r’ keyword. My mistake.
BUT THEN, it is possible in Python to enclose an instruction such as ?\3 which means (as far as I understood what you explained to me earlier) recursive reference to a regexp named ‘3’) ??? The ‘3’ name being given in the expression regexes[(3, (255,255,255))] is that correct ? SO you can reference an expression within itself while it has not be closed yet: the last parenthese of the expression 3 is after the \3). Is that what it means ?

Python syntax is a bit complicated to me.

[Alan Kilborn]

@Gilles-Maisonneuve said:

Python syntax is a bit complicated to me

It’s not Python syntax, it’s regular expression syntax. It’s just not Perl regular expression syntax. :)

And, BTW, nobody in the history of the world, especially someone coming from a Perl background, has ever uttered the phrase you typed.

[guy038]

Hello @gilles-maisonneuve, @eko-palypse and All,

Gilles, could you verify that the two lines, below, work, with yours Red, Green and Blue colors ?

regexes[(3, (R,G,B))] = (r'(?s-i)(<<)(['"]?)(\w+?)\2\h*;.*?\3', [1])
regexes[(4, (R,G,B))] = (r'(?s-i)(<<)\h+('|")(\w+?)\2\h*;.*?\3', [1,3])

For these two regexes :

• Group 1 = << = double inferior than sign

• Group 2 = ['"]? = an optional single or double quote, for regex id 3

• Group 2 = '|" = a mandatory single or double quote ,separated from the << characters with blank characters, for regex id 4

• Group 3 = \w+? = the shortest area of word characters, after the << sign, between possible quotes
  and before a semicolon character ;, with possible blank characters, before and/or after the quote characters

Notes :

• In regex id 3, only the << string is highlighted ( Group 1 )

• In regex id 4, the << and the text between quotes are highlighted ( Groups 1 and 3 )

• I added the -i in-line modifier ( => (?s-i) leading syntax ) to be sure that the ending boundary of the block corresponds exactly with the text, between quotes ( search is sensitive to case ! )

---

So my regex (?s-i)(<<)(['"]?)(\w+?)\2\h*;.*?\3 ( id 3 ) matches any of these six cases, below :

$x=<<TEXT;
Plain text here
TEXT

$x=<<'TEXT';
Plain text here
TEXT

$x=<<"TEXT";
Plain text here
TEXT

$x=<<TEXT ;
Plain text here
TEXT

$x=<<'TEXT' ;
Plain text here
TEXT

$x=<<"TEXT" ;
Plain text here
TEXT

And my regex (?s-i)(<<)\h+('|")(\w+?)\2\h*;.*?\3 ( id 4 ) matches these 4 cases, below :

$x=<< 'TEXT';
Plain text here
TEXT

$x=<< "TEXT";
Plain text here
TEXT

$x=<< 'TEXT' ;
Plain text here
TEXT

$x=<< "TEXT" ;
Plain text here
TEXT

Best Regards,

guy038

[Gilles Maisonneuve]

@guy038

Hello Guy,

Could not make it work, sorry.

I mean:

• added (replaced original ones) in the EnhancePerlLexer.py from Ekopalypse the following lines (according to what you gave me:

  regexes[(3, (224,0,0))] = (r’(?s-i)(<<)([‘"]?)(\w+?)\2\h*;.?\3’, [1])
  regexes[(4, (0,0,224))] = (r’(?s-i)(<<)\h+('|")(\w+?)\2\h;.*?\3’, [1,3])

• saved it and restarted npp

• list itemstill have the same coloring, not working.

BUT, good news:

python console:
Traceback (most recent call last):
File "C:\Users\gm\AppData\Roaming\Notepad++\plugins\Config\PythonScript\scripts\startup.py", line 1, in <module>
    import EnhancePerlLexer
File "C:\Users\gm\AppData\Roaming\Notepad++\plugins\Config\PythonScript\scripts\EnhancePerlLexer.py", line 36
    regexes[(3, (224,0,0))] = (r'(?s-i)(<<)(['"]?)(\w+?)\2\h*;.*?\3', [1])
                                                                        ^
SyntaxError: EOL while scanning string literal
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)]
Initialisation took 110ms
Ready.

Can you tell me what did I did wrong ?
(When I comment out the two lines I get back a valid coloring for the ‘q*’ syntaxes (yes, forgot to tell you, this had vanished too…)

[Gilles Maisonneuve]

@guy038

Well, I commented out the rule 3 and kept rule 4.
Same kind of error:

 regexes[(4, (0,0,224))] = (r'(?s-i)(<<)\h+('|")(\w+?)\2\h*;.*?\3', [1,3])
                                                                        ^
 SyntaxError: EOL while scanning string literal

[Gilles Maisonneuve]

if I modify the rule like:

regexes[(4, (0,0,224))] = (r'(?s-i)((<<)\h+([\'"])(\w+?)\2\h*;.*?\3)', [1,3])

I don’t get any longer a syntax error in Python BUT I get no coloring for the here doc either…

Any idea ?

[Gilles Maisonneuve]

@Alan-Kilborn

chcp 1250 >NUL: & perl -e "$var=q(Alan Kilborn est déplaisant dans sa façon de s'exprimer mais il a raison.); for my $p ('\t','\s') {print qq{\$p=$p},$var=~m/($p)déplaisant\1/x?$var:qq{n'en déplaise},qq{\n} ;};" & chcp 850 >NUL:

$p=\tn'en déplaise
$p=\sAlan Kilborn est déplaisant dans sa façon de s'exprimer mais il a raison.

[Gilles Maisonneuve]

J’ai tellement l’habitude d’utiliser $1, $2, …, qui, eux, ne fonctionnent pas dans un simple ‘match’ mais uniquement dans un ‘substitute’, que je ne connaissais pas cette façon de répéter les ‘patterns’ de ‘matching’. J’ai appris quelque chose.
Dont acte.

[Ekopalypse]

@Gilles-Maisonneuve

Lunch break :-)

First, I’m sorry not to telling you that the single quote has to be escaped as it was
used to denote a python string - good, you figured it already out.

Let me break down the parts of that python code

regexes = OrderedDict()
regexes[(3, (255,0,0))] = (r'(?s)(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)', [0])

regexes is variable, containing an OrderedDict class instance.
OrderedDict is more or less the same as a perl associative array or hash

regexes[] is the python way to access a key in that hash, like in perl regexes{}
regexes[()] the round bracket denotes a python tuple, in perl a list I guess (immutable)
the python tuple contains the items 3 and (255,0,0) <- this is again a tuple
The number 3 is here to create an unique key - has nothing to do with the regex itself.
So, regexes[(3, (255,0,0))] means, get me the value for key (3, (255,0,0)) from dict(hash) regexes

The value is (r’(?s)(\s*(<<)\s*(“{0,1}.+”{0,1})\s*;.*?\3)‘, [0])
Again, a python tuple containing the items r’…’ (raw string) and a list [] (in perl an array = mutable)
Everything within the raw string is the regex to be searched for and the list contains the information
which match group should be used for coloring
[0] is always the overall match of the complete regex and [1] would be the result from group 1,
[2] from group 2 and [1,2] from group 1 and group 2

So, in terms of regular expressions only the value part of the regexes hash/dict is of interest.
For searching only the raw string and for coloring which part was defined in the list [].

Does this makes sense to you?

The reason why this regex

regexes[(4, (0,0,224))] = (r'(?s-i)((<<)\h+([\'"])(\w+?)\2\h*;.*?\3)', [1,3])

doesn’t do what you want is that you use 4 groups now whereas @guy038 has
removed the outer matching group brackets.

(?s-i)(<<)(['"]?)(\w+?)\2\h*;.*?\3

In order to make it work either use

regexes[(4, (0,0,224))] = (r'(?s-i)(<<)\h+([\'"])(\w+?)\2\h*;.*?\3', [1,3])
or
regexes[(4, (0,0,224))] = (r'(?s-i)((<<)\h+([\'"])(\w+?)\3\h*;.*?\4)', [1,3])

[Alan Kilborn]

No idea what the “chcp 1250…” posting was supposed to be saying to me. :)

This thread gets my vote for the biggest jumbled mess in the history of the community. :)