Perl language syntax highlighting troubles (bug or limitation ?)
-
@guy038
Thanks.
I picked your 4th regex with END of here doc coloring included.
Just changed once again the \h+ in \h* to allow for no space between previous keyword or variable and the ‘<<’ operator.I think I’m going to use the regexp and python scripts provided in this thread to enhanced other syntax hilighting (I think about CMD and Yori for example. It’ll make me progress in Python, now that I have tested and tasted it.
Thanks to all again.
Gilles
-
Hello Eko,
I met the following shortcoming with the q* coloring:
use subs qw(divide_by_hand resultline is_fine_resultline resultline_len fine_resultline_len);
works fine, but if I want to split my qw() in 2 lines (because is becoming to be too long on the right side of screen), then I do:use subs qw(divide_by_hand resultline is_fine_resultline resultline_len fine_resultline_len);and I loose the coloring.
Could it be possible with python to include CRLF/NL (\n) in the pattern so that I can fold my qw (and even my other q*) statements ?
-
Hello Gilles,
how about using the single-line modifier
(?s)in front of the regex?
r'(?s)\bq[rwqx]{0,1}\b\h*(\(.+?\)|\[.+?\]|\{.+?\})'This should do the job, I assume.
Concerning the idea of using the script for multiple languages at the same time.
Currently the issue is that a second script would overwrite the variables
likeregexes,BUILTIN_LEXER,EnhanceBuiltinLexeretc… which would break
the logic of the first script.
My first thought would be ensuring namespace integrity, which means the script
needs to get modified in a way that no code gets be executed while importing the script. Which then would result in something like
import EnhancePerlLexerto import the script and an additional
EnhancePerlLexer.start()to activate it. Other scripts would then be configured likeimport EnhanceCmdLexerandEnhanceCmdLexer.start()etc…Another way would be to make a base class and overwrite it … hmm… need to think about it. Will try to find a way which is reasonable and easy to adapt.
Will keep you informed.Have a nice Sunday and greetings
Eren -
Hello Eren,
Worked like a charm (the
(?s)syntax), thank you.About the multiple language script… well I did not get your point exactly beside the fact that it is a lot more tricky than I thought. So, lazy as I am to learn Python, I’ll wait for you to find a solution to put in your EnhancePerlLexer.py (which then would become EnhanceAnyLexer.py).
Whenever you have time…
Thanks for all.
Gilles
-
Changed the regexp from:
regexes[(1, (128,0,128))] = (r'(?s)\bq[rwqx]{0,1}\b([^\h]).*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])
to:
..........................................................vv
regexes[(1, (128,0,128))] = (r'(?s)\bq[rwqx]{0,1}\b([^\h])\b.*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0])so that
qw( my words here are colored but not the remaing text after the closing parenthese, until it finds another one...);Without the word boundary when I used
"qw("instead of"qw ("I got the coloring extending past the closing)(respectively{...},[..], etc.)I know that this regex does not fit the full Perl syntax capability (any char separator like in
@array = qw xfoo bar quux;->@array = ('foo','bar','quu');) but I only use common separators like/{[()]}/. I leave it to someone else to figure this out… -
Hello, @gilles-maisonneuve,
If you decide to use
\h*, instead of\h+, in regex#4, then, some configurations can be found, either, with the regexes#3and#4! So, to be rigorous, you could use, other regexes, which define two disjoint sets :-
Here-doc documents, with TEXT, right after the
<<operator ( regex#3) -
Here-doc documents, with TEXT, after the
<<operator and possible space chars and a mandatory delimiter'or'( regex#4)
So, in the Python script :
# Color every "here-document" with the USER color r = 255 g = 0 b = 255 ( => Magenta ) regexes[(3, (255,0,255))] = (r'(?s-i)(<<)(\w+)\h*;.*?\2', [1]) regexes[(4, (255,0,255))] = (r'(?s-i)(<<)\h*(\x27|")(?|(\w+)\\?([\x27"]\w+)|(\w+)())\2\h*;.*?\3\4', [1,3,4])Tested, with success, against the Perl file “
Test_2_Gilles.pl”, recapitulating all cases, below :#----- NO SPACE nor " nor ', AFTER << ----- Colored with Regex #3 ----- $x=<<TEXT; Plain text here TEXT $x=<<TEXT; Plain text here TEXT #----- DELIMITER " or ', AFTER << ----- Colored with Regex #4 ----- $x=<<"TEXT"; Plain text here TEXT $x=<<'TE"XT' ; Plain text here TE"XT $x=<<"TE'XT" ; Plain text here TE'XT $x=<<'TE\'XT'; Plain text here TE'XT $x=<<"TE\"XT"; Plain text here TE"XT #----- SPACE(S) + DELIMITER " or ', AFTER << ----- Colored with Regex #4 ----- $x=<< 'TEXT'; Plain text here TEXT $x=<< "TEXT"; Plain text here TEXT $x=<< 'TE"XT'; Plain text here TE"XT $x=<< "TE'XT"; Plain text here TE'XT $x=<< 'TE\'XT'; Plain text here TE'XT $x=<< "TE\"XT"; Plain text here TE"XT #------------------------- END --------------------------Cheers,
guy038
-
-
I’m testing the EnhanceAnyBuiltinLexer right now and I noticed a situation where I don’t know exactly how to solve it better.
The issue is the following.
Assume that the text, which should get colored, are multiple lines. When the start or the end of that text is not within the visible area then the whole text is not colored.
I assume that is what @guy038 was mentioning.
:-(
To avoid this I use a sliding window approach (term borrowed from networks :-), which means as long as the first visible line is not greater than 15 the algorithm starts from line 0 and afterwards it is current line minus 15. Similar the end line gets added 15 unless the maximum lines would be reached.Not nice, but works as long as there is no text, which should be colored, contains more than 15 lines.
Any ideas how I could solve it differently?
Here the current code - maybe this makes it easier to understand what I’m trying to solve.
start_line = editor.docLineFromVisible(editor.getFirstVisibleLine()) end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen()) start_line -= 15 if start_line > 15 else 0 max_line = editor.getLineCount() if editor.getWrapMode(): end_line = sum([editor.wrapCount(x) for x in range(end_line)]) end_line += 15 if max_line - 15 > end_line else max_line start_position = editor.positionFromLine(start_line) end_position = editor.getLineEndPosition(end_line) editor.setIndicatorCurrent(self.INDICATOR_ID) editor.indicatorClearRange(0, editor.getTextLength()) for color, regex in self.regexes.items(): editor.research(regex[0], lambda match: self.paint_it(color[1], regex[1], match), 0, start_position, end_position)Thank you.
-
You have a good approach, I think. The only really great solution, that is great in one way and bad in another, is to lex over the entire document constantly. Obviously the problem with that is that it could take a great deal of time.
In keeping with your current approach, have you considered what happens when lines are folded or hidden? It seems that this would alter how far you need to “reach” beyond the current line (in both directions).
Pythonscript/Scintilla has
editor.docLineFromVisible()which may be of some use for this situation. Notepad++ itself uses it as well, see https://github.com/notepad-plus-plus/notepad-plus-plus/blob/master/PowerEditor/src/ScitillaComponent/SmartHighlighter.cpp#L83-L104 -
thank you very much for your insight, very much appreciated.
Obviously the problem with that is that it could take a great deal of time.
Right, I tested with a 5000 lines of code script and there was a noticeable delay.
In keeping with your current approach, have you considered what happens when lines are folded or hidden?
Yes, I’ve tested different scenarios when upper part has folded/hidden text or
within current visible area and bottom but neither of those are re-calculated,
so, you are right - this can become another issue.
What if hundreds of lines are folded but within the calculated range?
Need to test this more thoroughly.Pythonscript/Scintilla has editor.docLineFromVisible()
:-D yes, - looks like the first two lines of the example :-)
Again, thank you very much.
-
I’ve played with that for nearly two days and it seems to work.
Folding with ~10000 lines within the visible area seems to work without
any noticeable delay but a test with 20000 lines revealed already its weakness.The issue about having multiple lines colored is still open, the workaround
extends the visible area by 15 lines. If this should be changed, then you have to
change this lineself.offset_line = 15 # hack - see style function for more info
But be warned, the performance of the script comes from the fact that it tries to color
only the visible area every time an updateui event is fired and not by styling
the whole script every time.The following should be copied into a script, let’s call it
EnhanceAnyBuiltinLexer.py# -*- coding: utf-8 -*- from Npp import editor, editor1, editor2, notepad, NOTIFICATION, SCINTILLANOTIFICATION, INDICATORSTYLE, INDICFLAG from collections import OrderedDict from itertools import groupby, count SC_INDICVALUEBIT = 0x1000000 class SingletonEnhanceLexer(type): ''' Ensures, more or less, that only one instance of the main class can be instantiated ''' _instance = None def __call__(cls, *args, **kwargs): if cls._instance is None: cls._instance = super(SingletonEnhanceLexer, cls).__call__(*args, **kwargs) return cls._instance class EnhanceLexer(object): ''' Provides additional color options and should be used in conjunction with the built-in lexers. An indicator is used to avoid style collisions. Although the Scintilla documentation states that indicators 0-7 are reserved for the lexers, indicator 0 is used because none of the builtin lexers uses it internally. Actually, it looks like python lexer is the only lexer at all which uses an idicator and its value is 1 Even when using more than one regex, it is not necessary to define more than one indicator because the class uses the flag SC_INDICFLAG_VALUEFORE. See https://www.scintilla.org/ScintillaDoc.html#Indicators for more information on that topic ''' __metaclass__ = SingletonEnhanceLexer def __init__(self): ''' Instantiates the class, defines global indicator settings, registers callbacks and initializes needed variables. Because of __metaclass__ = ... usage, is called once only. ''' editor1.indicSetStyle(0, INDICATORSTYLE.TEXTFORE) editor1.indicSetFlags(0, INDICFLAG.VALUEFORE) editor2.indicSetStyle(0, INDICATORSTYLE.TEXTFORE) editor2.indicSetFlags(0, INDICFLAG.VALUEFORE) editor.callbackSync(self.on_updateui, [SCINTILLANOTIFICATION.UPDATEUI]) editor.callbackSync(self.on_marginclick, [SCINTILLANOTIFICATION.MARGINCLICK]) notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED]) notepad.callback(self.on_bufferactivated, [NOTIFICATION.BUFFERACTIVATED]) self.INDICATOR_ID = 0 self.registered_lexers = dict() self.doc_is_of_interest = False self.lexer_name = '' self.regexes = OrderedDict() self.excluded_styles = [] self.offset_line = 15 # hack - see style function for more info def register_lexer(self, lexer_name, _regexes, excluded_styles): ''' reformat provided regexes and cache everything within registered_lexers dictionary. Args: lexer_name = string, expected values as returned by editor.getLexerLanguage() _regexes = OrderedDict either in form of _regexes[(int, int)] = (r'', [int]) or _regexes[(int, (r, g, b))] = (r'', [int]) excluded_styles = list of integers Returns: None ''' self.lexer_name = lexer_name.lower() for k, v in _regexes.items(): if isinstance(k[1], tuple): fg_color = k[1] else: fg_color = editor.styleGetFore(k[1]) self.regexes[(k[0], self.rgb(*fg_color) | SC_INDICVALUEBIT)] = v self.registered_lexers[self.lexer_name] = (self.regexes, excluded_styles) def check_lexer(self): ''' Checks if the current document is of interest. If it is, loads the cached regexes and exclusion list. Sets the document flag accordingly Args: None Returns: None ''' self.lexer_name = editor.getLexerLanguage().lower() if self.lexer_name in self.registered_lexers: self.regexes, self.excluded_styles = self.registered_lexers[self.lexer_name] self.doc_is_of_interest = True else: self.doc_is_of_interest = False @staticmethod def rgb(r, g, b): ''' Helper function Retrieves rgb color triple and converts it into its integer representation Args: r = integer, red color value in range of 0-255 g = integer, green color value in range of 0-255 b = integer, blue color value in range of 0-255 Returns: integer ''' return (b << 16) + (g << 8) + r def paint_it(self, color, matchgroups, match): ''' This is where the actual coloring takes place. Color, matchgroups and match object must be provided. Matchgroups define which group(s) is(are) of interest Coloring occurs only if the position is not within the excluded range. Args: color = integer, expected in range of 0-16777215 matchgroups = list of integers match = python re.match object Returns: None ''' for group in matchgroups: start_pos = match.span(group)[0] if start_pos < 0 or editor.getStyleAt(start_pos) in self.excluded_styles: continue editor.setIndicatorCurrent(self.INDICATOR_ID) editor.setIndicatorValue(color) editor.indicatorFillRange(start_pos, match.span(group)[1] - start_pos) def style(self): ''' Calculates the text area to be searched for in the current document. Deletes the old indicators before setting new ones, searches for the defined regexes and calls colorize function on match. Args: None Returns: None ''' # TODO: # In cases where it is needed to color multiple lines there is a glitch # which affects coloring, if either start or end of that text is not visible. # Current workaround extends the visible area by self.offset_line lines, # which obviously won't work if more than self.offset_line lines need to be colored. # Need to find a better way to make sure that those parts do get colored as well. # # Maybe doing some kind of background caching and recalculate # only when there are changes in the document? start_line = editor.docLineFromVisible(editor.getFirstVisibleLine()) end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen()) start_line -= self.offset_line if start_line > self.offset_line else 0 max_line = editor.getLineCount() if editor.getWrapMode(): end_line = sum([editor.wrapCount(x) for x in range(end_line)]) end_line += self.offset_line if max_line - self.offset_line > end_line else max_line if editor.getAllLinesVisible(): groups = [(start_line, end_line)] else: visible_lines = [x for x in range(start_line, end_line) if editor.getLineVisible(x)] groups = [tuple(x) for _, x in groupby(visible_lines, key=lambda n, c=count(): n - next(c))] editor.setIndicatorCurrent(self.INDICATOR_ID) editor.indicatorClearRange(0, editor.getTextLength()) for group in groups: start_position = editor.positionFromLine(group[0]) end_position = editor.getLineEndPosition(group[-1]) for color, regex in self.regexes.items(): editor.research(regex[0], lambda match: self.paint_it(color[1], regex[1], match), 0, start_position, end_position) def on_bufferactivated(self, args): ''' Callback which gets called every time one switches a document. Triggers the check if the document is of interest. Args: provided by notepad object but none are of interest Returns: None ''' self.check_lexer() def on_updateui(self, args): ''' Callback which gets called every time scintilla (aka the editor) changed something within the document. Triggers the styling function if the document is of interest. Args: provided by scintilla but none are of interest Returns: None ''' if self.doc_is_of_interest: self.style() def on_langchanged(self, args): ''' Callback gets called every time one uses the Language menu to set a lexer Triggers the check if the document is of interest Args: provided by notepad object but none are of interest Returns: None ''' self.check_lexer() def on_marginclick(self, args): ''' Callback which gets called every time one clicks into the margin columns. margin 2 is the one of interest Triggers the styling function if the document is of interest. Args: margin returns an integer of the currently used column Returns: None ''' if args['margin'] == 2 and self.doc_is_of_interest: self.style() def main(self): ''' Main function entry point. Simulates two events to enforce detection of current document and potential styling. Args: None Returns: None ''' self.on_bufferactivated(None) self.on_updateui(None)Then, within either the user startup.py or any other new script the following should be
copied into.
Note, I haven’t updated the perl regexes so those might not be the one you want to use.# -*- coding: utf-8 -*- from collections import OrderedDict import EnhanceAnyBuiltinLexer enhance_lexer = EnhanceAnyBuiltinLexer.EnhanceLexer() # perl definitions starts regexes = OrderedDict() regexes[(1, 5)] = (r'\bq[rwqx]{0,1}\b([^\h]).*?\1|(\bq[rwqx]{0,1}\b\h+(\w).*?\3)', [0]) regexes[(2, 5)] = (r'(?s)\bq[rwqx]{0,1}\b\h*(\(.+?\)|\[.+?\]|\{.+?\})', [0]) regexes[(3, (255,0,0))] = (r'(?s)(\s*(<<)\s*("{0,1}.+"{0,1})\s*;.*?\3)', [0]) excluded_styles = [1, 2] enhance_lexer.register_lexer('perl', regexes, excluded_styles) # perl definitions ends # python definitions starts regexes = OrderedDict() regexes[(0, (224, 108, 117))] = (u'\\b(cls|self)\\b', [0]) regexes[(1, (209, 154, 102))] = (u'(?:(?:def)\s\w+)\s*\((.+)\):', [1]) regexes[(2, (86, 182, 194))] = (u'(\*|\*\*)(?=\w)', [0]) regexes[(3, (79, 175, 239))] = (u'class\s*\w+?(?=\()|def\s*\w+?(?=\()|(\w+?(?=\())', [1]) regexes[(4, (86, 182, 194))] = (u'\\b(editor|editor1|editor2|notepad|console|__init__|__call__|__del__|super|object|type|print)\\b', [0]) excluded_styles = [1, 3, 4, 6, 7, 12] enhance_lexer.register_lexer('python', regexes, excluded_styles) # python definitions ends # # put additional definitions for other lexers here # # should be last statement # first current document doesn't trigger bufferactivated # therefore call main to simulate it enhance_lexer.main() -
For those being directed here by other links from the forum, @Ekopalypse has a newer version available at his github, here.
