Community
    • Login

    Perl language syntax highlighting troubles (bug or limitation ?)

    Scheduled Pinned Locked Moved Help wanted · · · – – – · · ·
    112 Posts 6 Posters 43.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Meta ChuhM
      Meta Chuh moderator @Alan Kilborn
      last edited by

      maybe @Ekopalypse will write a resuming manual, once this is over … i refuse :)

      1 Reply Last reply Reply Quote 3
      • EkopalypseE
        Ekopalypse
        last edited by

        You mean a short manager summary I guess :-D

        Meta ChuhM 1 Reply Last reply Reply Quote 1
        • Meta ChuhM
          Meta Chuh moderator @Ekopalypse
          last edited by

          if a short manager summary is, in your eyes, a fully featured guide, covering all eventualities, based on all caveats of the whole topic … then yes 😉

          1 Reply Last reply Reply Quote 3
          • EkopalypseE
            Ekopalypse
            last edited by

            LOL - back to business

            1 Reply Last reply Reply Quote 2
            • Gilles MaisonneuveG
              Gilles Maisonneuve @Ekopalypse
              last edited by Gilles Maisonneuve

              @Ekopalypse
              Replying at the message “Lunch break…” with all the explainations.

              1. THANK YOU ! I start enjoying Python since I read you.
                Well, I’ll never be a disciple, because not fan of OO and have difficulty to accept a language where the tabulations and spaces define the code syntax… reminds me too much of my youth with the punch cards and the punched paper roll (and yes, I’m that old), but it’s kind of fun to read when one understand it better.
                Your analogies with Perl made it very comprehensible, very kind of you.

              2. ** Y Y Y Y E E E E S S S S ! ! ! ! ! ! **
                It works. I just changed a ‘+’ into a ‘*’ after the first ‘\h’ to allow for no horizontal space between the preceding keyword / Perl-separator and the here doc starter (‘<<’). So my (yours with my ridiculous pinch of salt) regexp is now:

                regexes[(4, (0,0,224))] = (r’(?s-i)((<<)\h*([‘"])(\w+?)\3\h*;.*?\4)’, [1,3])
                ^^

              Thank to you and your patience (and for the readers, pissed at my garbage, this includes you Alan, and BTW ignore my perl joke, I was upset by the tone you used and perhaps, even further by the fact that you were right [chcp needed to get the accents on the French vowels on a Windows Perl console]).

              I was without-a-clue and you saved my day. And now…

              Already several adventures have begun to take shape which can be solved by no-one else. Right, Eko ?
              Right you are, Meta Chuh.
              And so, without further ado… …I hereby declare this case… …closed.

              {to sum up the solution provided by Ekopalypse in this thread of discussion}:

              regexes[(1, (255,0,128))] = (r'\bq[rwqx]{0,1}\b([^\h]).*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])
              regexes[(2, (255,0,128))] = (r'\bq[rwqx]{0,1}\b\h*(\(.+?\)|\[.+?\]|\{.+?\})', [0])
              regexes[(4, (0,112,112))] = (r'(?s-i)((<<)\h*([\'"])(\w+?)\3\h*;.*?\4)', [1,3])
              

              Allow you to colorize your Perl ‘q*’ keywords and args with color RGB 255,0,128 and your here-docs with color RGB 0,112,112 in Notepad-plus-plus, using “Python Script” plugin. By so, you correct the coloring limitation of Scintilla Library for those keywords in Perl.

              Hope this summary will be satisfying enough for “Meta Chuh al.”.

              Have nice week (end of) all of you.

              Gilles

              Alan KilbornA EkopalypseE 2 Replies Last reply Reply Quote 3
              • Alan KilbornA
                Alan Kilborn @Gilles Maisonneuve
                last edited by

                @Gilles-Maisonneuve said:

                pissed at my garbage, this includes you Alan

                Oh, not at all…at least after 98 messages a positive outcome!

                1 Reply Last reply Reply Quote 1
                • EkopalypseE
                  Ekopalypse @Gilles Maisonneuve
                  last edited by Ekopalypse

                  @Gilles-Maisonneuve

                  Thank you for your kind words.
                  I have enjoyed developing the script with you and the
                  discussion afterwards was also helpful and intersting
                  as it pointed out that the script comments aren’t 100% bulletproof.
                  And please don’t hesitate to post here if you find something that
                  doesn’t work the way you expected it - just let us know, in the end
                  we can all benefit from it.

                  Regarding the off topic comments, these were not meant to be insulting.
                  Sometimes there is a sound in it that is only understood if you read through
                  several other answers - they are mostly nice but sometimes they do provoke
                  but they are still meant to be nice or at least helpful.
                  The one with the summaries, for example, was from another thread where
                  I caused a confusion, because of my recklessness, that was only cleared up
                  a few posts later and then I gave a “manager summary” to among other things,
                  to make life easier for future readers. Seems to become a running gag now :-)

                  I agree with you, Python is in the beginning odd, especially
                  when coming from a different language.
                  It took some time getting used to it and I had my difficulties too,
                  but now I find the language super - especially Python3, which is
                  by the way not supported by the PythonScript plugin, :-(
                  has syntax constructs and language extensions that I really like.

                  So, have a nice weekend too.

                  Gilles MaisonneuveG 1 Reply Last reply Reply Quote 2
                  • guy038G
                    guy038
                    last edited by guy038

                    Hello @gilles-maisonneuve, @eko-palypse, @meta-chuh, @alan-kilborn, @peterjones and All,

                    Ah ! So I’m going to do the 101th post ;-)) Don’t worry, I won’t be [too] long ! I will :

                    • Explain why my previous regexes did not work ( almost obvious )

                    • Give you a new version of all the regexes, used in the EkoPalypse script, which :

                      • Matches the case qq|qr|qw|qx|q with the < and > delimiters

                      • Matches the case of here-docs, containing an escaped delimiter ( \' or \" ), inside the starting and ending blocks ( legal syntax )

                    For information, refer to :

                    https://perldoc.perl.org/perlop.html

                    And particularly :

                    https://perldoc.perl.org/perlop.html#Quote-Like-Operators

                    https://perldoc.perl.org/perlop.html#Quote-and-Quote-like-Operators


                    So, Gilles, in my previous regexes, I used the single quote symbol as it ! At that moment I just tested the regular expressions, without using the Python script :-((

                    Once I decided to use the Eko’s script, I quickly understood that the single quote symbol is, first, interpreted by the Python engine. So, when I changed any ' single quote with the syntax \x27, in my previous regexes, everything went OK ;-))


                    Then, I decided to test the m Perl instruction, first, with all possible delimiters ( See my .pl test file, below )

                    For instance :

                    m bPATTERNb
                    m ZPATTERNZ
                    m 0PATTERN0
                    m _PATTERN_
                    m(PATTERN)
                    m<PATTERN>
                    m [PATTERN]
                    m {PATTERN}
                    m!PATTERN!
                    m"PATTERN"
                    m/PATTERN/
                    m $PATTERN$
                    m %PATTERN%
                    m &PATTERN&
                    ....
                    ....
                    

                    On the same way, I tried all syntaxes of the PERL instruction qrDelimiterPATTERNDelimiter So, in regex #1, relative to the q PERL instructions, I enumerated all possible delimiters, different from a word char and from the four sets () [] {} <>, in the character class [!"#$%&\x27*+,./:;=?@``|~\\^-]

                    Note that in that regex #1, I used the special syntax (?|........|.......|... ....|....), which forces the renumbering of the groups, located inside the group, for each alternative ! ( See an example, at the end of that post )

                    Then, I tried to enumerate all the variations of the here-docs syntax, including special cases as, for instance

                    $x=<< "TE\"XT";
                    Plain text here
                    TE"XT
                    

                    And I succeeded to manage this case in my new regexes #3 and #4 ;-))

                    However, note that the highlighting of any here-document is effective ONLY IF the ending text is visible, in the current editor window !


                    So, here are my new regexes :

                    # Color every instruction word qq|qr|qw|qx|q with PERL Style 5 ( r = 0 v = 0 b = 255 => Blue )
                    
                    regexes[(1, 5)] = (r'(?s-i)\bq[qrwx]?(?|\h*([!"#$%&\x27*+,./:;=?@`|~\\^-])|\h+(\w)).*?\1', [0])
                    regexes[(2, 5)] = (r'(?s-i)\bq[qrwx]?\h*(\(.+?\)|\[.+?\]|\{.+?\}|<.+?>)', [0])
                    
                    # Color every here-document with the USER color r = 255 g = 0 b = 255 ( => Magenta )
                    
                    regexes[(3, (255,0,255))] = (r'(?s-i)(<<)([\x27"]?)(?|(\w+)\\([\x27"]\w+)|(\w+)())\2\h*;.*?\3\4', [1])
                    regexes[(4, (255,0,255))] = (r'(?s-i)(<<)\h+(\x27|")(?|(\w+)\\([\x27"]\w+)|(\w+)())\2\h*;.*?\3\4', [1,3,4])
                    
                    # If, on addition, you want to highlight the END of here-docs :
                    
                    #regexes[(3, (255,0,255))] = (r'(?s-i)(<<)([\x27"]?)(?|(\w+)\\([\x27"]\w+)|(\w+)())\2\h*;.*?(\3)(\4)', [1,5,6])
                    #regexes[(4, (255,0,255))] = (r'(?s-i)(<<)\h+(\x27|")(?|(\w+)\\([\x27"]\w+)|(\w+)())\2\h*;.*?(\3)(\4)', [1,3,4,5,6])
                    

                    If you want to know how these regexes work, I could give you some hints, next time. Just too lazy to do it, right now ;-))

                    Note also, that I added, in comments, regexes #3 and #4 if you want, also, highlight the end of here-docs, by placing the back-references \3 and \4, inside parentheses => Two new groups 5 and 6

                    Remark : Do not delete the empty group () in regexes #3 and #4 : it represents an empty group 4, re-used by the back-reference \4


                    And, of course, here is, below, the Test_Gilles.pl file, used to test these 4 new regexes :

                    #--------------------------------------------------------------------------------------
                    
                    # Various examples of INSTRUCTION WORDS q, qq, qr, qw, qx, highlighted with the PYTHON script
                    
                    q/ok/error    q(ok);
                    qrw/ok/error    q(ok);
                    qq/ok/error;   qq{ok};
                    qr/ok/error;   qr(ok);
                    qw/ok/error;   qw[ok];
                    qq/ok/error;   qx(ok);
                    
                    q        xokxerror      q (ok);
                    qq     hokherror;     qq {ok};
                    qr                      rokerror;    qr  (ok);
                    qw aokaerror;   qw [ok];
                    qx    zokzerror;   qx (ok);
                    
                    #--------------------------------------------------------------------------------------
                    
                    my var1 = q xfoobarx;
                    my var2 = q getservbyname g;
                    my var3 = q getservbyname getservbyent;
                    my var4 = qx{ verify > NUL: };
                    my var5 = qr/$singer.*grand chanteur/;
                    
                    #--------------------------------------------------------------------------------------
                    
                    $bar = q(\n);              # or   $bar = '\n'
                    
                    $foo = qq(\n);             # or   $bar = "\n" ( Interpolation )
                    
                    $abc = qx(echo .);         # or   $abc = `echo .`
                    
                    $perl_info  = qx(ps $$);   # That's Perl's $$
                    $shell_info = qx'ps $$';   # That's the new shell's $$
                    
                    use POSIX qw( setlocale localeconv )
                    @EXPORT = qw( foo bar baz );
                    
                    qr/PATTERN/msixpodualn     # Interpolation occurs unless delimiter is a SINGLE quote '
                    
                    #--------------------------------------------------------------------------------------
                    
                    $r = qr//;
                    $rex = qr/my.STRING/is;
                    $re = qr/$pattern/;
                    qr/$_/i
                    
                    next if qr#^/usr/spool/uucp# ;
                    
                    #--------------------------------------------------------------------------------------
                    
                     my $sentence_rx = qr&
                       (?: (?<=^) | (?<=\s) )  # after start-of-string or # whitespace
                       \p{Lu}                  # capital letter
                       .*?                     # a bunch of anything
                       [.?!]                   # followed by a sentence ender
                       (?= $ | \s )            # in front of end-of-string or whitespace
                      &sx;
                    
                    #--------------------------------------------------------------------------------------
                    
                    The cases where WHITESPACE must be used are when the QUOTING character is a WORD character :
                    
                    q XfooX                    # Means the string 'foo'
                    qx XfooX                   # Means the string 'foo', too
                    
                    qXfooX                     # WRONG !
                    qxXfooX                    # WRONG !
                    
                    #--------------------------------------------------------------------------------------
                    
                    # There can (and in some cases, must) be WHITESPACE between the operator and
                    # the quoting characters, EXCEPT when # is being used as the quoting character :
                    #
                    # q#foo# is parsed as the string foo , while q #foo# is the operator q followed by a
                    # comment. So, its argument will be taken from the next line.
                    
                    q#foo#
                    
                    q #foo#   #  => ONLY q SHOULD be colored ( Exception )
                    
                    #--------------------------------------------------------------------------------------
                    
                    # Instruction Word m, already CORRECTLY highlighted, by DEFAULT, by Scintilla :
                    
                    mBPATTERNB   # KO ( normal )
                    mZPATTERNZ   # KO ( normal )
                    mbPATTERNb   # KO ( normal )
                    mzPATTERNz   # KO ( normal )
                    m0PATTERN0   # KO ( normal )
                    m9PATTERN9   # KO ( normal )
                    m_PATTERN_   # KO ( normal )
                    
                    m BPATTERNB
                    m ZPATTERNZ
                    m zPATTERNz
                    m 0PATTERN0
                    m 9PATTERN9
                    m _PATTERN_
                    
                    m(PATTERN)
                    m<PATTERN>
                    m[PATTERN]
                    m{PATTERN}
                    
                    m (PATTERN)
                    m <PATTERN>
                    m [PATTERN]
                    m {PATTERN}
                    
                    m!PATTERN!
                    m"PATTERN"
                    m#PATTERN#
                    m$PATTERN$
                    m%PATTERN%
                    m&PATTERN&
                    m'PATTERN'
                    m*PATTERN*
                    m+PATTERN+
                    m,PATTERN,
                    m-PATTERN-
                    m.PATTERN.
                    m/PATTERN/
                    m:PATTERN:
                    m;PATTERN;
                    m=PATTERN=
                    m?PATTERN?
                    m@PATTERN@
                    m\PATTERN\
                    m^PATTERN^
                    m`PATTERN`
                    m|PATTERN|
                    m~PATTERN~
                    
                    m !PATTERN!
                    m "PATTERN"
                    m #PATTERN#
                    m $PATTERN$
                    m %PATTERN%
                    m &PATTERN&
                    m 'PATTERN'
                    m *PATTERN*
                    m +PATTERN+
                    m ,PATTERN,
                    m -PATTERN-
                    m .PATTERN.
                    m /PATTERN/
                    m :PATTERN:
                    m ;PATTERN;
                    m =PATTERN=
                    m ?PATTERN?
                    m @PATTERN@
                    m \PATTERN\
                    m ^PATTERN^
                    m `PATTERN`
                    m |PATTERN|
                    m ~PATTERN~
                    
                    #--------------------------------------------------------------------------------------
                    
                    # Instruction Word qr, CORRECTLY highlighted, with the PYTHON script :
                    
                    qrBPATTERNB    # KO ( normal )
                    qrZPATTERNZ    # KO ( normal )
                    qrbPATTERNb    # KO ( normal )
                    qrzPATTERNz    # KO ( normal )
                    qr0PATTERN0    # KO ( normal )
                    qr9PATTERN9    # KO ( normal )
                    qr_PATTERN_    # KO ( normal )
                    
                    qr BPATTERNB
                    qr ZPATTERNZ
                    qr bPATTERNb
                    qr zPATTERNz
                    qr 0PATTERN0
                    qr 9PATTERN9
                    qr _PATTERN_
                    
                    qr(PATTERN)
                    qr<PATTERN>
                    qr[PATTERN]
                    qr{PATTERN}
                    
                    qr (PATTERN)
                    qr <PATTERN>
                    qr [PATTERN]
                    qr {PATTERN}
                    
                    qr!PATTERN!
                    qr"PATTERN"
                    qr#PATTERN#
                    qr$PATTERN$
                    qr%PATTERN%
                    qr&PATTERN&
                    qr'PATTERN'
                    qr*PATTERN*
                    qr+PATTERN+
                    qr,PATTERN,
                    qr-PATTERN-
                    qr.PATTERN.
                    qr/PATTERN/
                    qr:PATTERN:
                    qr;PATTERN;
                    qr=PATTERN=
                    qr?PATTERN?
                    qr@PATTERN@
                    qr\PATTERN\
                    qr^PATTERN^
                    qr`PATTERN`
                    qr|PATTERN|
                    qr~PATTERN~
                    
                    qr !PATTERN!
                    qr "PATTERN"
                    qr #PATTERN#
                    qr $PATTERN$
                    qr %PATTERN%
                    qr &PATTERN&
                    qr 'PATTERN'
                    qr *PATTERN*
                    qr +PATTERN+
                    qr ,PATTERN,
                    qr -PATTERN-
                    qr .PATTERN.
                    qr /PATTERN/
                    qr :PATTERN:
                    qr ;PATTERN;
                    qr =PATTERN=
                    qr ?PATTERN?
                    qr @PATTERN@
                    qr \PATTERN\
                    qr ^PATTERN^
                    qr `PATTERN`
                    qr |PATTERN|
                    qr ~PATTERN~
                    
                    #--------------------------------------------------------------------------------------
                    
                    # For completeness, << as shift operator
                    
                    $b = (1 << 5);
                    
                    #  Here-documents, CORRECTLY highlighted, with the PYTHON script :
                    
                    $x=<<TEXT;
                    Plain text here
                    TEXT
                    
                    $x=<<'TEXT';
                    Plain text here
                    TEXT
                    
                    $x=<<"TEXT";
                    Plain text here
                    TEXT
                    
                    $x=<<'TE"XT';
                    Plain text here
                    TE"XT
                    
                    $x=<<"TE'XT";
                    Plain text here
                    TE'XT
                    
                    # Here-documents, with a SPACE char, before the SEMI-COLON
                    
                    $x=<<TEXT ;
                    Plain text here
                    TEXT
                    
                    $x=<<'TEXT' ;
                    Plain text here
                    TEXT
                    
                    $x=<<"TEXT" ;
                    Plain text here
                    TEXT
                    
                    $x=<<'TE"XT' ;
                    Plain text here
                    TE"XT
                    
                    $x=<<"TE'XT" ;
                    Plain text here
                    TE'XT
                    
                    
                    #  Here-documents, with the ESCAPED delimiter in the TEXT, CORRECTLY highlighted, too !
                    
                    $x=<<'TE\'XT';
                    Plain text here
                    TE'XT
                    
                    $x=<<"TE\"XT";
                    Plain text here
                    TE"XT
                    
                    # The SAME + a SPACE char, before the SEMI-COLON
                    
                    $x=<<'TE\'XT' ;
                    Plain text here
                    TE'XT
                    
                    $x=<<"TE\"XT" ;
                    Plain text here
                    TE"XT
                    
                    # Here-documents with SPACE highlights as operator, in Notepad++
                    
                    $x=<< 'TEXT';
                    Plain text here
                    TEXT
                    
                    $x=<< "TEXT";
                    Plain text here
                    TEXT
                    
                    $x=<< 'TE"XT';
                    Plain text here
                    TE"XT
                    
                    $x=<< "TE'XT";
                    Plain text here
                    TE'XT
                    
                    # The SAME + a SPACE char, before the SEMI-COLON
                    
                    $x=<< 'TEXT' ;
                    Plain text here
                    TEXT
                    
                    $x=<< "TEXT" ;
                    Plain text here
                    TEXT
                    
                    $x=<< 'TE"XT' ;
                    Plain text here
                    TE"XT
                    
                    $x=<< "TE'XT" ;
                    Plain text here
                    TE'XT
                    
                    #  Here-docs with SPACE highlights as operator, and the ESCAPED delimiter in TEXT, CORRECTLY highlighted !
                    
                    $x=<< 'TE\'XT';
                    Plain text here
                    TE'XT
                    
                    $x=<< "TE\"XT";
                    Plain text here
                    TE"XT
                    
                    # The SAME + a SPACE char, before the SEMI-COLON
                    
                    $x=<< 'TE\'XT' ;
                    Plain text here
                    TE'XT
                    
                    $x=<< "TE\"XT" ;
                    Plain text here
                    TE"XT
                    
                    
                    #-----  Note that MULTIPLE Here-docs are NOT managed, yet -:(( -----
                    
                    print <<"foo", <<"bar"; # you can stack them
                    I said foo.
                    foo
                    I said bar.
                    bar
                    
                    myfunc(<< "THIS", 23, <<'THAT');
                    Here's a line
                    or two.
                    THIS
                    and here's another.
                    THAT
                    
                    #------------------ END ----------------------------
                    

                    Cheers,

                    guy038

                    P.S :

                    Here a simple example of the (?|......|.......|.......)

                    Let’s suppose that you want to match these two expressions :

                    foo12345fooABCDE
                    bar12345barABCDE

                    A classic syntax should be (foo)12345\1ABCDE|(bar)12345\2ABCDE, where group 1 = foo and group 2 = bar

                    But you can use this second shorter regex (?|(foo)|(bar))12345\1ABCDE, where group 1 represents, either, foo or bar, depending of the part of the alternative has matched

                    For a more complete example, refer to :

                    https://www.boost.org/doc/libs/1_55_0/libs/regex/doc/html/boost_regex/syntax/perl_syntax.html#boost_regex.syntax.perl_syntax.branch_reset

                    This (?|PATTERN) syntax is, commonly, called a branch-reset !

                    Gilles MaisonneuveG 1 Reply Last reply Reply Quote 3
                    • Gilles MaisonneuveG
                      Gilles Maisonneuve @guy038
                      last edited by

                      @guy038
                      Thanks.
                      I picked your 4th regex with END of here doc coloring included.
                      Just changed once again the \h+ in \h* to allow for no space between previous keyword or variable and the ‘<<’ operator.

                      I think I’m going to use the regexp and python scripts provided in this thread to enhanced other syntax hilighting (I think about CMD and Yori for example. It’ll make me progress in Python, now that I have tested and tasted it.

                      Thanks to all again.

                      Gilles

                      1 Reply Last reply Reply Quote 2
                      • Gilles MaisonneuveG
                        Gilles Maisonneuve @Ekopalypse
                        last edited by

                        @Ekopalypse

                        Hello Eko,

                        I met the following shortcoming with the q* coloring:
                        use subs qw(divide_by_hand resultline is_fine_resultline resultline_len fine_resultline_len);
                        works fine, but if I want to split my qw() in 2 lines (because is becoming to be too long on the right side of screen), then I do:

                        use subs qw(divide_by_hand resultline is_fine_resultline
                                    resultline_len fine_resultline_len);
                        

                        and I loose the coloring.

                        Could it be possible with python to include CRLF/NL (\n) in the pattern so that I can fold my qw (and even my other q*) statements ?

                        EkopalypseE 1 Reply Last reply Reply Quote 1
                        • EkopalypseE
                          Ekopalypse @Gilles Maisonneuve
                          last edited by

                          @Gilles-Maisonneuve

                          Hello Gilles,

                          how about using the single-line modifier (?s) in front of the regex?
                          r'(?s)\bq[rwqx]{0,1}\b\h*(\(.+?\)|\[.+?\]|\{.+?\})'

                          This should do the job, I assume.

                          Concerning the idea of using the script for multiple languages at the same time.
                          Currently the issue is that a second script would overwrite the variables
                          like regexes, BUILTIN_LEXER, EnhanceBuiltinLexer etc… which would break
                          the logic of the first script.
                          My first thought would be ensuring namespace integrity, which means the script
                          needs to get modified in a way that no code gets be executed while importing the script. Which then would result in something like
                          import EnhancePerlLexer to import the script and an additional
                          EnhancePerlLexer.start() to activate it. Other scripts would then be configured like

                          import EnhanceCmdLexer and EnhanceCmdLexer.start() etc…

                          Another way would be to make a base class and overwrite it … hmm… need to think about it. Will try to find a way which is reasonable and easy to adapt.
                          Will keep you informed.

                          Have a nice Sunday and greetings
                          Eren

                          Gilles MaisonneuveG 1 Reply Last reply Reply Quote 2
                          • Gilles MaisonneuveG
                            Gilles Maisonneuve @Ekopalypse
                            last edited by

                            @Ekopalypse

                            Hello Eren,

                            Worked like a charm (the (?s) syntax), thank you.

                            About the multiple language script… well I did not get your point exactly beside the fact that it is a lot more tricky than I thought. So, lazy as I am to learn Python, I’ll wait for you to find a solution to put in your EnhancePerlLexer.py (which then would become EnhanceAnyLexer.py).

                            Whenever you have time…

                            Thanks for all.

                            Gilles

                            Gilles MaisonneuveG 1 Reply Last reply Reply Quote 1
                            • Gilles MaisonneuveG
                              Gilles Maisonneuve @Gilles Maisonneuve
                              last edited by

                              Changed the regexp from:
                              regexes[(1, (128,0,128))] = (r'(?s)\bq[rwqx]{0,1}\b([^\h]).*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])
                              to:
                              ..........................................................vv
                              regexes[(1, (128,0,128))] = (r'(?s)\bq[rwqx]{0,1}\b([^\h])\b.*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])

                              so that

                              qw( my words here
                                     are colored but not the remaing text after
                                     the closing parenthese, until it finds another one...);
                              

                              Without the word boundary when I used "qw(" instead of "qw (" I got the coloring extending past the closing ) (respectively {...}, [..], etc.)

                              I know that this regex does not fit the full Perl syntax capability (any char separator like in @array = qw xfoo bar quux; -> @array = ('foo','bar','quu');) but I only use common separators like /{[()]}/. I leave it to someone else to figure this out…

                              EkopalypseE 1 Reply Last reply Reply Quote 0
                              • guy038G
                                guy038
                                last edited by guy038

                                Hello, @gilles-maisonneuve,

                                If you decide to use \h*, instead of \h+, in regex #4, then, some configurations can be found, either, with the regexes #3 and #4 ! So, to be rigorous, you could use, other regexes, which define two disjoint sets :

                                • Here-doc documents, with TEXT, right after the << operator ( regex #3 )

                                • Here-doc documents, with TEXT, after the << operator and possible space chars and a mandatory delimiter ' or ' ( regex #4 )

                                So, in the Python script :

                                # Color every "here-document" with the USER color r = 255 g = 0 b = 255 ( => Magenta )
                                regexes[(3, (255,0,255))] = (r'(?s-i)(<<)(\w+)\h*;.*?\2', [1])
                                regexes[(4, (255,0,255))] = (r'(?s-i)(<<)\h*(\x27|")(?|(\w+)\\?([\x27"]\w+)|(\w+)())\2\h*;.*?\3\4', [1,3,4])
                                

                                Tested, with success, against the Perl file “Test_2_Gilles.pl”, recapitulating all cases, below :

                                #----- NO SPACE nor " nor ', AFTER << ----- Colored with Regex #3 -----
                                
                                $x=<<TEXT;
                                Plain text here
                                TEXT
                                
                                $x=<<TEXT;
                                Plain text here
                                TEXT
                                
                                #----- DELIMITER " or ', AFTER << ----- Colored with Regex #4 -----
                                
                                $x=<<"TEXT";
                                Plain text here
                                TEXT
                                
                                $x=<<'TE"XT' ;
                                Plain text here
                                TE"XT
                                
                                $x=<<"TE'XT" ;
                                Plain text here
                                TE'XT
                                
                                $x=<<'TE\'XT';
                                Plain text here
                                TE'XT
                                
                                $x=<<"TE\"XT";
                                Plain text here
                                TE"XT
                                
                                #----- SPACE(S) + DELIMITER " or ', AFTER << ----- Colored with Regex #4 -----
                                
                                $x=<< 'TEXT';
                                Plain text here
                                TEXT
                                
                                $x=<< "TEXT";
                                Plain text here
                                TEXT
                                
                                $x=<< 'TE"XT';
                                Plain text here
                                TE"XT
                                
                                $x=<< "TE'XT";
                                Plain text here
                                TE'XT
                                
                                $x=<< 'TE\'XT';
                                Plain text here
                                TE'XT
                                
                                $x=<< "TE\"XT";
                                Plain text here
                                TE"XT
                                
                                #------------------------- END --------------------------
                                

                                Cheers,

                                guy038

                                1 Reply Last reply Reply Quote 3
                                • EkopalypseE
                                  Ekopalypse
                                  last edited by

                                  I’m testing the EnhanceAnyBuiltinLexer right now and I noticed a situation where I don’t know exactly how to solve it better.
                                  The issue is the following.
                                  Assume that the text, which should get colored, are multiple lines. When the start or the end of that text is not within the visible area then the whole text is not colored.
                                  I assume that is what @guy038 was mentioning.
                                  :-(
                                  To avoid this I use a sliding window approach (term borrowed from networks :-), which means as long as the first visible line is not greater than 15 the algorithm starts from line 0 and afterwards it is current line minus 15. Similar the end line gets added 15 unless the maximum lines would be reached.

                                  Not nice, but works as long as there is no text, which should be colored, contains more than 15 lines.

                                  Any ideas how I could solve it differently?

                                  Here the current code - maybe this makes it easier to understand what I’m trying to solve.

                                          start_line = editor.docLineFromVisible(editor.getFirstVisibleLine())
                                          end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen())
                                  
                                          start_line -= 15 if start_line > 15 else 0
                                  
                                          max_line = editor.getLineCount()
                                          if editor.getWrapMode():
                                              end_line = sum([editor.wrapCount(x) for x in range(end_line)])
                                          end_line += 15 if max_line - 15 > end_line else max_line
                                  
                                          start_position = editor.positionFromLine(start_line)
                                          end_position = editor.getLineEndPosition(end_line)
                                  
                                          editor.setIndicatorCurrent(self.INDICATOR_ID)
                                          editor.indicatorClearRange(0, editor.getTextLength())
                                          for color, regex in self.regexes.items():
                                              editor.research(regex[0],
                                                              lambda match: self.paint_it(color[1],
                                                                                          regex[1],
                                                                                          match),
                                                              0,
                                                              start_position,
                                                              end_position)
                                  

                                  Thank you.

                                  Alan KilbornA 1 Reply Last reply Reply Quote 2
                                  • Alan KilbornA
                                    Alan Kilborn @Ekopalypse
                                    last edited by

                                    @Ekopalypse

                                    You have a good approach, I think. The only really great solution, that is great in one way and bad in another, is to lex over the entire document constantly. Obviously the problem with that is that it could take a great deal of time.

                                    In keeping with your current approach, have you considered what happens when lines are folded or hidden? It seems that this would alter how far you need to “reach” beyond the current line (in both directions).

                                    Pythonscript/Scintilla has editor.docLineFromVisible() which may be of some use for this situation. Notepad++ itself uses it as well, see https://github.com/notepad-plus-plus/notepad-plus-plus/blob/master/PowerEditor/src/ScitillaComponent/SmartHighlighter.cpp#L83-L104

                                    EkopalypseE 1 Reply Last reply Reply Quote 4
                                    • EkopalypseE
                                      Ekopalypse @Alan Kilborn
                                      last edited by

                                      @Alan-Kilborn

                                      thank you very much for your insight, very much appreciated.

                                      Obviously the problem with that is that it could take a great deal of time.

                                      Right, I tested with a 5000 lines of code script and there was a noticeable delay.

                                      In keeping with your current approach, have you considered what happens when lines are folded or hidden?

                                      Yes, I’ve tested different scenarios when upper part has folded/hidden text or
                                      within current visible area and bottom but neither of those are re-calculated,
                                      so, you are right - this can become another issue.
                                      What if hundreds of lines are folded but within the calculated range?
                                      Need to test this more thoroughly.

                                      Pythonscript/Scintilla has editor.docLineFromVisible()

                                      :-D yes, - looks like the first two lines of the example :-)

                                      Again, thank you very much.

                                      1 Reply Last reply Reply Quote 3
                                      • EkopalypseE
                                        Ekopalypse @Gilles Maisonneuve
                                        last edited by

                                        @Gilles-Maisonneuve

                                        I’ve played with that for nearly two days and it seems to work.
                                        Folding with ~10000 lines within the visible area seems to work without
                                        any noticeable delay but a test with 20000 lines revealed already its weakness.

                                        The issue about having multiple lines colored is still open, the workaround
                                        extends the visible area by 15 lines. If this should be changed, then you have to
                                        change this line self.offset_line = 15 # hack - see style function for more info
                                        But be warned, the performance of the script comes from the fact that it tries to color
                                        only the visible area every time an updateui event is fired and not by styling
                                        the whole script every time.

                                        The following should be copied into a script, let’s call it EnhanceAnyBuiltinLexer.py

                                        # -*- coding: utf-8 -*-
                                        from Npp import editor, editor1, editor2, notepad, NOTIFICATION, SCINTILLANOTIFICATION, INDICATORSTYLE, INDICFLAG
                                        from collections import OrderedDict
                                        from itertools import groupby, count
                                        
                                        SC_INDICVALUEBIT = 0x1000000
                                        
                                        
                                        class SingletonEnhanceLexer(type):
                                                '''
                                                    Ensures, more or less, that only one
                                                    instance of the main class can be instantiated
                                                '''
                                        
                                                _instance = None
                                        
                                                def __call__(cls, *args, **kwargs):
                                                    if cls._instance is None:
                                                        cls._instance = super(SingletonEnhanceLexer, cls).__call__(*args, **kwargs)
                                                    return cls._instance
                                        
                                        
                                        class EnhanceLexer(object):
                                            '''
                                                Provides additional color options and should be used
                                                in conjunction with the built-in lexers.
                                                An indicator is used to avoid style collisions.
                                                Although the Scintilla documentation states that indicators 0-7 are
                                                reserved for the lexers, indicator 0 is used because none of the
                                                builtin lexers uses it internally.
                                                Actually, it looks like python lexer is the only lexer at all
                                                which uses an idicator and its value is 1
                                        
                                                Even when using more than one regex, it is not necessary
                                                to define more than one indicator because the class uses
                                                the flag SC_INDICFLAG_VALUEFORE.
                                                See https://www.scintilla.org/ScintillaDoc.html#Indicators
                                                for more information on that topic
                                            '''
                                            __metaclass__ = SingletonEnhanceLexer
                                        
                                            def __init__(self):
                                                '''
                                                    Instantiates the class, defines global indicator settings,
                                                    registers callbacks and initializes needed variables.
                                                    Because of __metaclass__ = ... usage, is called once only.
                                                '''
                                                editor1.indicSetStyle(0, INDICATORSTYLE.TEXTFORE)
                                                editor1.indicSetFlags(0, INDICFLAG.VALUEFORE)
                                                editor2.indicSetStyle(0, INDICATORSTYLE.TEXTFORE)
                                                editor2.indicSetFlags(0, INDICFLAG.VALUEFORE)
                                                editor.callbackSync(self.on_updateui, [SCINTILLANOTIFICATION.UPDATEUI])
                                                editor.callbackSync(self.on_marginclick, [SCINTILLANOTIFICATION.MARGINCLICK])
                                                notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED])
                                                notepad.callback(self.on_bufferactivated, [NOTIFICATION.BUFFERACTIVATED])
                                                self.INDICATOR_ID = 0
                                                self.registered_lexers = dict()
                                                self.doc_is_of_interest = False
                                                self.lexer_name = ''
                                                self.regexes = OrderedDict()
                                                self.excluded_styles = []
                                                self.offset_line = 15  # hack - see style function for more info
                                        
                                        
                                            def register_lexer(self, lexer_name, _regexes, excluded_styles):
                                                '''
                                        
                                                    reformat provided regexes and cache everything
                                                    within registered_lexers dictionary.
                                        
                                                    Args:
                                                        lexer_name = string, expected values as returned by editor.getLexerLanguage()
                                                        _regexes = OrderedDict either in form of
                                                                   _regexes[(int, int)] = (r'', [int]) or
                                                                   _regexes[(int, (r, g, b))] = (r'', [int])
                                                        excluded_styles = list of integers
                                                    Returns:
                                                        None
                                                '''
                                                self.lexer_name = lexer_name.lower()
                                                for k, v in _regexes.items():
                                                    if isinstance(k[1], tuple):
                                                        fg_color = k[1]
                                                    else:
                                                        fg_color = editor.styleGetFore(k[1])
                                        
                                                    self.regexes[(k[0], self.rgb(*fg_color) | SC_INDICVALUEBIT)] = v
                                        
                                                self.registered_lexers[self.lexer_name] = (self.regexes, excluded_styles)
                                        
                                        
                                            def check_lexer(self):
                                                '''
                                                    Checks if the current document is of interest.
                                                    If it is, loads the cached regexes and exclusion list.
                                                    Sets the document flag accordingly
                                        
                                                    Args:
                                                        None
                                                    Returns:
                                                        None
                                                '''
                                                self.lexer_name = editor.getLexerLanguage().lower()
                                                if self.lexer_name in self.registered_lexers:
                                                    self.regexes, self.excluded_styles = self.registered_lexers[self.lexer_name]
                                                    self.doc_is_of_interest = True
                                                else:
                                                    self.doc_is_of_interest = False
                                        
                                        
                                            @staticmethod
                                            def rgb(r, g, b):
                                                '''
                                                    Helper function
                                                    Retrieves rgb color triple and converts it
                                                    into its integer representation
                                        
                                                    Args:
                                                        r = integer, red color value in range of 0-255
                                                        g = integer, green color value in range of 0-255
                                                        b = integer, blue color value in range of 0-255
                                                    Returns:
                                                        integer
                                                '''
                                                return (b << 16) + (g << 8) + r
                                        
                                        
                                            def paint_it(self, color, matchgroups, match):
                                                '''
                                                    This is where the actual coloring takes place.
                                                    Color, matchgroups and match object must be provided.
                                                    Matchgroups define which group(s) is(are) of interest
                                                    Coloring occurs only if the position is not within the excluded range.
                                        
                                                    Args:
                                                        color = integer, expected in range of 0-16777215
                                                        matchgroups = list of integers
                                                        match = python re.match object
                                                    Returns:
                                                        None
                                                '''
                                                for group in matchgroups:
                                                    start_pos = match.span(group)[0]
                                        
                                                    if start_pos < 0 or editor.getStyleAt(start_pos) in self.excluded_styles:
                                                        continue
                                        
                                                    editor.setIndicatorCurrent(self.INDICATOR_ID)
                                                    editor.setIndicatorValue(color)
                                                    editor.indicatorFillRange(start_pos, match.span(group)[1] - start_pos)
                                        
                                        
                                            def style(self):
                                                '''
                                                    Calculates the text area to be searched for in the current document.
                                                    Deletes the old indicators before setting new ones, searches for
                                                    the defined regexes and calls colorize function on match.
                                        
                                                    Args:
                                                        None
                                                    Returns:
                                                        None
                                                '''
                                                # TODO:
                                                # In cases where it is needed to color multiple lines there is a glitch
                                                # which affects coloring, if either start or end of that text is not visible.
                                                # Current workaround extends the visible area by self.offset_line lines,
                                                # which obviously won't work if more than self.offset_line lines need to be colored.
                                                # Need to find a better way to make sure that those parts do get colored as well.
                                                #
                                                # Maybe doing some kind of background caching and recalculate
                                                # only when there are changes in the document?
                                        
                                                start_line = editor.docLineFromVisible(editor.getFirstVisibleLine())
                                                end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen())
                                        
                                                start_line -= self.offset_line if start_line > self.offset_line else 0
                                        
                                                max_line = editor.getLineCount()
                                                if editor.getWrapMode():
                                                    end_line = sum([editor.wrapCount(x) for x in range(end_line)])
                                        
                                                end_line += self.offset_line if max_line - self.offset_line > end_line else max_line
                                        
                                                if editor.getAllLinesVisible():
                                                    groups = [(start_line, end_line)]
                                                else:
                                                    visible_lines = [x for x in range(start_line, end_line) if editor.getLineVisible(x)]
                                                    groups = [tuple(x) for _, x in groupby(visible_lines, key=lambda n, c=count(): n - next(c))]
                                        
                                                editor.setIndicatorCurrent(self.INDICATOR_ID)
                                                editor.indicatorClearRange(0, editor.getTextLength())
                                        
                                                for group in groups:
                                                    start_position = editor.positionFromLine(group[0])
                                                    end_position = editor.getLineEndPosition(group[-1])
                                        
                                                    for color, regex in self.regexes.items():
                                                        editor.research(regex[0],
                                                                        lambda match: self.paint_it(color[1],
                                                                                                    regex[1],
                                                                                                    match),
                                                                        0,
                                                                        start_position,
                                                                        end_position)
                                        
                                        
                                            def on_bufferactivated(self, args):
                                                '''
                                                    Callback which gets called every time one switches a document.
                                                    Triggers the check if the document is of interest.
                                        
                                                    Args:
                                                        provided by notepad object but none are of interest
                                                    Returns:
                                                        None
                                                '''
                                                self.check_lexer()
                                        
                                        
                                            def on_updateui(self, args):
                                                '''
                                                    Callback which gets called every time scintilla
                                                    (aka the editor) changed something within the document.
                                        
                                                    Triggers the styling function if the document is of interest.
                                        
                                                    Args:
                                                        provided by scintilla but none are of interest
                                                    Returns:
                                                        None
                                                '''
                                                if self.doc_is_of_interest:
                                                    self.style()
                                        
                                        
                                            def on_langchanged(self, args):
                                                '''
                                                    Callback gets called every time one uses the Language menu to set a lexer
                                                    Triggers the check if the document is of interest
                                        
                                                    Args:
                                                        provided by notepad object but none are of interest
                                                    Returns:
                                                        None
                                                '''
                                                self.check_lexer()
                                        
                                        
                                            def on_marginclick(self, args):
                                                '''
                                                    Callback which gets called every time one clicks into the margin columns.
                                                    margin 2 is the one of interest
                                        
                                                    Triggers the styling function if the document is of interest.
                                        
                                                    Args:
                                                        margin returns an integer of the currently used column
                                                    Returns:
                                                        None
                                                '''
                                                if args['margin'] == 2 and self.doc_is_of_interest:
                                                    self.style()
                                        
                                        
                                            def main(self):
                                                '''
                                                    Main function entry point.
                                                    Simulates two events to enforce detection of current document
                                                    and potential styling.
                                        
                                                    Args:
                                                        None
                                                    Returns:
                                                        None
                                                '''
                                                self.on_bufferactivated(None)
                                                self.on_updateui(None)
                                        

                                        Then, within either the user startup.py or any other new script the following should be
                                        copied into.
                                        Note, I haven’t updated the perl regexes so those might not be the one you want to use.

                                        # -*- coding: utf-8 -*-
                                        from collections import OrderedDict
                                        import EnhanceAnyBuiltinLexer
                                        enhance_lexer = EnhanceAnyBuiltinLexer.EnhanceLexer()
                                        
                                        # perl definitions starts
                                        regexes = OrderedDict()
                                        regexes[(1, 5)] = (r'\bq[rwqx]{0,1}\b([^\h]).*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])
                                        regexes[(2, 5)] = (r'(?s)\bq[rwqx]{0,1}\b\h*(\(.+?\)|\[.+?\]|\{.+?\})', [0])
                                        regexes[(3, (255,0,0))] = (r'(?s)(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)', [0])
                                        excluded_styles = [1, 2]
                                        enhance_lexer.register_lexer('perl', regexes, excluded_styles)
                                        # perl definitions ends
                                        
                                        
                                        # python definitions starts
                                        regexes = OrderedDict()
                                        regexes[(0, (224, 108, 117))] = (u'\\b(cls|self)\\b', [0])
                                        regexes[(1, (209, 154, 102))] = (u'(?:(?:def)\s\w+)\s*\((.+)\):', [1])
                                        regexes[(2, (86, 182, 194))]  = (u'(\*|\*\*)(?=\w)', [0])
                                        regexes[(3, (79, 175, 239))]  = (u'class\s*\w+?(?=\()|def\s*\w+?(?=\()|(\w+?(?=\())', [1])
                                        regexes[(4, (86, 182, 194))]  = (u'\\b(editor|editor1|editor2|notepad|console|__init__|__call__|__del__|super|object|type|print)\\b', [0])
                                        excluded_styles = [1, 3, 4, 6, 7, 12]
                                        enhance_lexer.register_lexer('python', regexes, excluded_styles)
                                        # python definitions ends
                                        
                                        # 
                                        # put additional definitions for other lexers here
                                        #
                                        
                                        # should be last statement
                                        # first current document doesn't trigger bufferactivated 
                                        # therefore call main to simulate it
                                        enhance_lexer.main()
                                        
                                        PeterJonesP 1 Reply Last reply Reply Quote 2
                                        • PeterJonesP
                                          PeterJones @Ekopalypse
                                          last edited by

                                          For those being directed here by other links from the forum, @Ekopalypse has a newer version available at his github, here.

                                          1 Reply Last reply Reply Quote 1
                                          • First post
                                            Last post
                                          The Community of users of the Notepad++ text editor.
                                          Powered by NodeBB | Contributors