Using 'Enhance UDL Lexer' script to configure the syntax for multiple languages

[tyler durden]

After a lot of trial and error, I’ve found a solution!! :D

I do not know if it’s the best method or the most efficient, but it’s certainly getting the job done. Now, I can call a single script to make all the changes I want (in all the User Defined Languages).

It’d be awesome to get a second opinion. Please let me know what mistakes I made (situations where this script might fail, inefficiency, etc…).

Here it is:

This first snippet really made my day. I used lambda functions to match the pattern inside the configuration area. So, if you follow the pattern below, you can use its predictability to compute everything. I only tested this with two languages (Go and Python), but I added different styles to both to make sure everything works. This should work on more UDLs and/or more styles, but I’m not 100% sure.

# --------------------- CONFIGURATION AREA --------------------- #

# LANGUAGE,
# R, G, B, REGEX, MATCH_GROUP

"Go_Lang",
189, 147, 249, "\d\.\d+", 0,
139, 233, 253, "(\w+)\(", 1,

"Py_Lang",
189, 147, 249, "\d\.\d+", 0,
120, 221, 84, "(\w+)\(", 1,
240, 143, 47, "\@\w+", 0,
252,196,153,"\.(\w+)",1,
195,250,195,"^class(.\w*)",1

# --------------------- CONFIGURATION AREA --------------------- #

Note:

• "Go_Lang" and "Py_Lang" are the exact names (matching upper/lowercase) of my UDLs in the Languages tab.
• UDL names must be defined once – no more, no less. After that, you can just describe the style for each respective language.
• The style configuration should follow the original script by @Ekopalypse with a few minor changes: [1] do not add the r that precedes the regex byte string; [2] do not add a unique number for each style configuration, this is done automatically; [3] remember to add a comma , after each and every item, except for the last one.

Here’s the full script:

# -*- coding: utf-8 -*-

from Npp import editor, editor1, editor2, notepad, NOTIFICATION, SCINTILLANOTIFICATION, INDICATORSTYLE
import ctypes
import ctypes.wintypes as wintypes
from collections import OrderedDict


set_up_list = [ \

# --------------------- CONFIGURATION AREA --------------------- #

# LANGUAGE,
# R, G, B, REGEX, MATCH_GROUP

"Go_Lang",
189, 147, 249, "\d\.\d+", 0,
139, 233, 253, "(\w+)\(", 1,

"Py_Lang",
189, 147, 249, "\d\.\d+", 0,
120, 221, 84, "(\w+)\(", 1,
240, 143, 47, "\@\w+", 0,
252,196,153,"\.(\w+)",1,
195,250,195,"^class(.\w*)",1

# --------------------- CONFIGURATION AREA --------------------- #

]#end of set_up_list's square brackets

def language_set_up(set_up_list):
	all_regexes = {}
	regexes = {}
	pattern = lambda x,k: 5*x + k
	
	row_count = 0
	keep_going = True

	while keep_going == True:
		try:
			LANGUAGE = set_up_list[0]
			ID = row_count
			R = set_up_list[pattern(row_count,1)]
			G = set_up_list[pattern(row_count,2)]
			B = set_up_list[pattern(row_count,3)]
			REGEX = r"%s" % set_up_list[pattern(row_count,4)]
			MATCH_GROUP = set_up_list[pattern(row_count,5)]
			regexes[(ID, (R,G,B))] = (REGEX,MATCH_GROUP)
			
			row_count += 1

			if type(set_up_list[pattern(row_count,1)]) == str:
				all_regexes[LANGUAGE] = regexes
				del set_up_list[0:pattern(row_count,1)]
				regexes = {}
				row_count = 0
		except IndexError:
			keep_going = False
	all_regexes[LANGUAGE] = regexes	
	return all_regexes

regexes = language_set_up(set_up_list)

excluded_styles = [1, 2, 16, 17, 18, 19, 20, 21, 22, 23]

user32 = wintypes.WinDLL("user32")
WM_USER = 1024
NPPMSG = WM_USER+1000
NPPM_GETLANGUAGEDESC = NPPMSG+84
SC_INDICVALUEBIT = 0x1000000
SC_INDICFLAG_VALUEFORE = 1

class EnhanceUDLLexer:
	
	def __init__(self, current_regex, current_language):
		self.current_regex = current_regex
		self.current_language = current_language
		editor.callbackSync(self.on_updateui, [SCINTILLANOTIFICATION.UPDATEUI])
		notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED])
		notepad.callback(self.on_bufferactivated, [NOTIFICATION.BUFFERACTIVATED])
		self.doc_is_of_interest = False
		self.lexer_name = None
		self.npp_hwnd = user32.FindWindowW(u"Notepad++", None)
		self.configure()

	@staticmethod
	def rgb(r, g, b):
		return (b << 16) + (g << 8) + r


	@staticmethod
	def paint_it(color, pos, length):
		if pos < 0 or editor.getStyleAt(pos) in excluded_styles:
			return
		editor.setIndicatorCurrent(0)
		editor.setIndicatorValue(color)
		editor.indicatorFillRange(pos, length)

	def style(self):
		start_line = editor.docLineFromVisible(editor.getFirstVisibleLine())
		end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen())
		start_position = editor.positionFromLine(start_line)
		end_position = editor.getLineEndPosition(end_line)
		editor.setIndicatorCurrent(0)
		editor.indicatorClearRange(0, editor.getTextLength())
		for color, regex in self.regexes.items():
			editor.research(regex[0],
							lambda m: self.paint_it(color[1],
													m.span(regex[1])[0],
													m.span(regex[1])[1] - m.span(regex[1])[0]),
							0,
							start_position,
							end_position)
	def configure(self):
		editor1.indicSetStyle(0, INDICATORSTYLE.TEXTFORE)
		editor1.indicSetFlags(0, SC_INDICFLAG_VALUEFORE)
		editor2.indicSetStyle(0, INDICATORSTYLE.TEXTFORE)
		editor2.indicSetFlags(0, SC_INDICFLAG_VALUEFORE)
		self.regexes = OrderedDict([ ((k[0], self.rgb(*k[1]) | SC_INDICVALUEBIT), v) for k, v in self.current_regex.items() ])
		self.lexer_name = u"User Defined language file - %s" % self.current_language

	def check_lexer(self):
		language = notepad.getLangType()
		length = user32.SendMessageW(self.npp_hwnd, NPPM_GETLANGUAGEDESC, language, None)
		buffer = ctypes.create_unicode_buffer(u' ' * length)
		user32.SendMessageW(self.npp_hwnd, NPPM_GETLANGUAGEDESC, language, ctypes.byref(buffer))
		self.doc_is_of_interest = True if buffer.value == self.lexer_name else False

	def on_bufferactivated(self, args):
		self.check_lexer()


	def on_updateui(self, args):
		if self.doc_is_of_interest:
			self.style()

	def on_langchanged(self, args):
		self.check_lexer()


	def main(self):
		self.on_bufferactivated(None)
		self.on_updateui(None)


for key in list(regexes.keys()):
	EnhanceUDLLexer(regexes[key],key).main()

I’ll try to make improvements to this script. I thought about other things like adding bold/ underline/italic, changing background color, compiling the script to .pyc or .exe to optimize, getting rid of the .xml style file and do everything via script to literally control every bit of the text editor, prevent tab lag (it happens sometimes). However, I don’t know if these things are possible or if I’m able to do it. If anyone has insight on these topics or the script itself, it’d be great to hear about it.

Thank you!

[Alan Kilborn]

@tyler-durden said in Using 'Enhance UDL Lexer' script to configure the syntax for multiple languages:

compiling the script to .pyc or .exe to optimize

That one doesn’t make any sense in the context, but I’ll let @Ekopalypse comment on the other stuff as he is the author of the original.

[tyler durden]

@Alan-Kilborn yeah…I thought that this might be a bit weird. I’ve used the NppExec plugin in the past to do some simple stuff, so I was curious to know if that sort of stuff would be possible with or without the NppExec plugin.

[Alan Kilborn]

@tyler-durden

So a bit of background. NppExec can certainly run Python, but the environment you’d get isn’t the one you need for the task at hand. You need something that is aware of the Notepad++ objects, specifically the editor object. This relates to the import stuff you were asking about before.

[Ekopalypse]

Ohhh - this changed more than I thought it might do but good to see
that you already started to improve it - well done.

I just was about to post the link to the repo with the modified version

Regarding the question about Npp object, this is the object
PythonScript itself defines within its cpp code. It uses boost::python to
create the python objects.

I will have a look at your version and see if I can “borrow” some code :-)

[Ekopalypse]

@tyler-durden @Alan-Kilborn

I never timed a compiled version with a non-compiled version but I
assume that the difference isn’t that big if there is a difference at all.
I’ll try to measure this.

@tyler-durden

With the newest PythonScript version 1.5.4 there is no need to use
ctypes in order to get the language name anymore. You can use
notepad.getLanguageName(notepad.getLangType())

A performance issue might arise for this part of the code

for key in list(regexes.keys()):
	EnhanceUDLLexer(regexes[key],key).main()

This will lead to multiple callbacks calls. Every new language will
create another set of callbacks and all do get called even only one language is used at a time.
Switch the regexes on a bufferactivated event.

[Ekopalypse]

@tyler-durden @Alan-Kilborn

I did the test with PythonScript3 and a compiled (pyc) and a non-compiled version.

The result is that both are more or less equally fast.
I used the script itself to measure how long it takes to format the code in an update ui event.
Result: 0.8 - 2.0 milliseconds on my machine.
Sometimes the non-compiled version was faster, next time the pyc version was faster.

[tyler durden]

@Ekopalypse
I followed your instructions regarding the use of ctypes and the execution of multiple callbacks. I’ve also read your EnhanceAnyLexer.py and “borrowed” a few things (a_few_things == 99% lol). I know it’s a bit pointless to do this since you’ve already wrote a great tool to control multiple lexers, but whatever…

Differences include:

• I don’t check if the version is 1.5.4.0 since I know my version,
• I don’t have a rgb(r, g, b) method – I do that outside the EnhanceUDLLexer class,
• I don’t register lexers – your advice about “switching the regexes on a bufferactivated event” was very helpful and now I just switch dictionaries inside the style method (self.all_regexes[self.current_language]) based on notepad.getLanguageName(notepad.getLangType()),
• I kept the configure method,
• and some other minor changes.

I guess the real difference is the input method for each lexer style since the set_up_list at the beginning gets all rgb’s, regexes, etc… Here’s the final version:

# -*- coding: utf-8 -*-
# needs Python Script version 1.5.4.0
from Npp import (notepad,editor,editor1,editor2,NOTIFICATION,
				SCINTILLANOTIFICATION,INDICATORSTYLE,INDICFLAG,INDICVALUE)

set_up_list = [ \

# --------------------- CONFIGURATION AREA --------------------- #

#  "LANGUAGE"
#  R, G, B, "REGEX", MATCH_GROUP
#  ...
#  ...
#  R, G, B, "REGEX", MATCH_GROUP
#  [excluded_style_0, excluded_style_1, ..., excluded_style_23]

"Go_Lang", ## UDL
189, 147, 249, r"\d\.\d+", 0,
139, 233, 253, r"(\w+)\(", 1,
[1, 4, 6, 7, 16, 17, 18, 19],

"Py_Lang", ## UDL
189, 147, 249, r"\d\.\d+", 0,
120, 221, 84, r"(\w+)\(", 1,
240, 143, 47, r"\@\w+", 0,
198, 95, 107, r"\.(\w+)", 1,
86, 182, 194,r"(\w+)\.", 1,
195, 250, 195, r"^class(.\w*)", 1,
[1, 2, 3, 4, 6, 7, 12, 16, 17, 18, 19],

"C_Lang", ## UDL
183, 252, 55, r"(\w+)\(", 1,
255, 121, 198, r"\#\w+", 0,
90, 100, 250, r"\b(stack|node)\b",0,
[1, 3, 4, 6, 7, 12, 16, 17, 18, 19],

"JavaScript", ## not an UDL
120, 221, 84, r"\((.*?)\)", 0,
90, 100, 250, r"\b(var|require)\b",0,
[1, 2, 3, 4, 6, 7, 12, 16, 17, 18, 19],

"HTML", ## not an UDL
90, 100, 250, r"\b(href)\b",0,
189, 147, 249, r"\d\.\d+", 0,
120, 221, 84, r"(\w+)\(", 1,
[0],

"CSS", ## not an UDL
120, 221, 84, r"(.*?)\{", 1,
255, 166, 85, r"[{}]", 0,
255, 84, 18, r"[:;]", 0,
[0],

"R_Lang", ## UDL
120, 221, 84, r"(\w+)\(", 1,
[1, 3, 4, 6, 7, 12, 16, 17, 18, 19], ## add comma at the end

# --------------------- CONFIGURATION AREA --------------------- #

None
]#end of set_up_list's square brackets

def language_set_up(set_up_list):
	all_regexes = {}
	excluded_styles = {}
	regexes = {}
	pattern = lambda x,k: 5*x + k
	change_rgb = lambda r,g,b: (b << 16) + (g << 8) + r

	row_count = 0

	while True:
		if isinstance(set_up_list[pattern(row_count,1)], list):
			LANGUAGE = set_up_list[0]
			all_regexes[LANGUAGE] = regexes
			excluded_styles[LANGUAGE] = set_up_list[pattern(row_count,1)]
			if set_up_list[pattern(row_count,2)] != None:
				del set_up_list[0:pattern(row_count,2)]
				regexes = {}
				row_count = 0
			else:
				break

		ID = row_count
		R = set_up_list[pattern(row_count,1)]
		G = set_up_list[pattern(row_count,2)]
		B = set_up_list[pattern(row_count,3)]
		REGEX = set_up_list[pattern(row_count,4)]
		MATCH_GROUP = set_up_list[pattern(row_count,5)]
		
		regexes[(ID, change_rgb(R,G,B) | INDICVALUE.BIT)] = (REGEX,MATCH_GROUP)
		
		row_count += 1

	all_regexes[LANGUAGE] = regexes	
	return all_regexes, excluded_styles

regexes, excluded_styles = language_set_up(set_up_list)

class EnhanceUDLLexer:
	
	def __init__(self, all_regexes, excluded_styles):
		self.INDICATOR_ID = 0
		self.doc_is_of_interest = False
		self.all_regexes = all_regexes
		self.excluded_styles = excluded_styles
		self.current_language = ""
		self.configure()

	def configure(self):
		editor1.indicSetStyle(self.INDICATOR_ID, INDICATORSTYLE.TEXTFORE)
		editor1.indicSetFlags(self.INDICATOR_ID, INDICFLAG.VALUEFORE)
		editor2.indicSetStyle(self.INDICATOR_ID, INDICATORSTYLE.TEXTFORE)
		editor2.indicSetFlags(self.INDICATOR_ID, INDICFLAG.VALUEFORE)
		editor.callbackSync(self.on_updateui, [SCINTILLANOTIFICATION.UPDATEUI])
		editor.callbackSync(self.on_marginclick, [SCINTILLANOTIFICATION.MARGINCLICK])
		notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED])
		notepad.callback(self.on_bufferactivated, [NOTIFICATION.BUFFERACTIVATED])

	def paint_it(self, color, match_position, length, start_position, end_position):
		if (match_position + length < start_position or
			match_position > end_position or
			editor.getStyleAt(match_position) in self.excluded_styles[self.current_language]):
			return

		editor.setIndicatorCurrent(0)
		editor.setIndicatorValue(color)
		editor.indicatorFillRange(match_position, length)


	def style(self):
		start_line = editor.docLineFromVisible(editor.getFirstVisibleLine())
		end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen())
		if editor.getWrapMode():
			end_line = sum([editor.wrapCount(x) for x in range(end_line)])

		onscreen_start_position = editor.positionFromLine(start_line)
		onscreen_end_pos = editor.getLineEndPosition(end_line)

		editor.setIndicatorCurrent(0)
		editor.indicatorClearRange(0, editor.getTextLength())
		for color, regex in self.all_regexes[self.current_language].items():
			editor.research(regex[0],
							lambda m: self.paint_it(color[1],
													m.span(regex[1])[0],
													m.span(regex[1])[1] - m.span(regex[1])[0],
													onscreen_start_position,
													onscreen_end_pos),
							0,
							onscreen_start_position,
							onscreen_end_pos)

	def check_lexer(self):
		current_language = notepad.getLanguageName(notepad.getLangType()).replace('udf - ','')
		if current_language in self.all_regexes:
			self.doc_is_of_interest = True
			self.current_language = current_language

	def on_marginclick(self, args):
		if args['margin'] == 2 and self.doc_is_of_interest:
			self.style()

	def on_bufferactivated(self, args):
		self.check_lexer()

	def on_updateui(self, args):
		if self.doc_is_of_interest:
			self.style()

	def on_langchanged(self, args):
		self.check_lexer()


	def main(self):
		self.on_bufferactivated(None)
		self.on_updateui(None)


Enhance = EnhanceUDLLexer(regexes,excluded_styles)
Enhance.main()

I timed this script, my old script (with multiple callbacks) and your EnhanceAnyLexer.py script. All of them were timed using the same method and with the same number of lexers (7 = 4 UDLs and 3 built-in lexers). I did not change the registration process when timing EnhanceAnyLexer.py. In other words, I added styles as expected, for example: r_regexes = _dict() => r_regexes[(0, (120, 221, 84))] = (r"(\w+)\(", 1) => r_excluded_styles = [1, 3, 4, 6, 7, 12, 16, 17, 18, 19].

I timed all scripts by adding a the following to them:

import time
start = time.time()
#
# actual lexer script
#
end = time.time()
print "Lexer Time: %.2f ms" % (1000*(end-start))

I don’t know if this method is accurate or appropriate for this particular situation (if you know of a better way, please let me know), but both EnhanceAnyLexer.py and my new script got an average of 30 ms, which seems to indicate that I don’t have any performance issues like multiple callbacks, etc. However, my old script, just as you said, got an average of 3250 ms lol … 7 lexers with multiple callbacks do add up quickly.

So, that’s it. I’m pretty happy with the result. Two things I’d like to ask are: [1] is it possible to control bold/italic/underline/other rich text stuff via Python? [2] instead of excluding entire styles, is it possible to excluded specific keywords/operators/delimiters/etc. like excluding only the word def for example.?

Thank you!!

[Ekopalypse]

@tyler-durden

I know it’s a bit pointless to do this …

I even think this makes sense if you want to extend/modify existing code.
Personally I learn fastest this way. Good job.

Measuring execution time is one of those things.
Actually you should make sure that only Npp and PythonScript(PS) are used
to make sure that other plugins don’t interfere.
Because of the accuracy I think that tests that take several milliseconds are relatively accurate,
unless you are programming for a RealTime system, but then Python is probably the wrong language as well.
Higher accuracy could be achieved with ctypes and highperformance counter,
but do we shoot sparrows with cannons here?
But if the question was about how to code timing functions, I prefer decorators.
I have a user startup.py file which is automatically executed on every startup, because it is called startup.py.
Please do not confuse it with the delivered startup.py from PS. If you make changes there, they would be lost when you update.
In this user startup.py I have several functions like

from time import perf_counter
def timeit(func):
    def wrapped(*args, **kwargs):
        s = perf_counter()
        x = func(*args, **kwargs)
        e = perf_counter()
        print(f'Calling {func.__name__} took {e-s}')
        return x
    return wrapped

for Python2 it must be changed to

from time import time
def timeit(func):
    def wrapped(*args, **kwargs):
        s = time()
        x = func(*args, **kwargs)
        e = time()
        print('Calling {} took {}'.format(func.__name__, e-s))
        return x
    return wrapped

Then all you need to do is

@timeit
def my_function_to_test(p1,p2):
    pass

I’m pretty happy with the result.

And that counts, script is readable, extendable … what would one else.

1. Yes and no. Scintilla knows two ways to colorize text, Styles and Indicators.
   Styles offer you the full range of possibilities scintilla offers to manipulate text attributes.
   Indicators can only change the font color. In general both can be manipulated from PS.
   The reason why indicators are used here is because Lexer uses styles.
   If we would also use styles we would always be in “clinch” with the lexers, whose definition is the final one.

2. Currently the script doesn’t give that, maybe use a regular expression that takes that into account?
   Or am I in the middle of something and don’t understand what you mean?

[tyler durden]

@Ekopalypse
First of all, thank you for the info about measuring execution time.

My lexer script is saved here: C:\Users\rtl\AppData\Roaming\Notepad++\plugins\config\PythonScript\scripts\EnhanceUDLLexer.py and to automatically execute it on every startup I added the following line import EnhanceUDLLexer at the end of the delivered startup.py from PythonScript (which is located here C:\Program Files (x86)\Notepad++\plugins\PythonScript\scripts\startup.py).

I imported the lexer script (EnhanceUDLLexer.py) from the delivered startup.py because I didn’t know you could have other startup.py scripts. Which path should I choose for my user startup file to make it run on every startup? Is it bad to alter the delivered startup.py like I did with the extra import?

About my previous questions:
[1] I think I understand the basics of this issues. That’s not a problem though…I asked just out of curiosity.

[2] I guess bulit-in lexers like HTML, CSS, JavaScript, etc. define their styles with large groups and a lot of keywords/operators etc. get grouped together in such a way that excluding one style may lead to losing the style on some other stuff that I didn’t want to lose style. But…now that I think about it, it might be easier to just copy the built-in stuff into an UDL and make all the changes I want. Never mind about this question.

Thanks again!

[Ekopalypse]

@tyler-durden

Which path should I choose for my user startup file to make it run on every startup?

Just create a new script and name it startup.py.
It will be automatically put into your user script directory.

Is it bad to alter the delivered startup.py like I did with the extra import?

The included startup.py will be replaced when there is a PythonScript update, otherwise, no.

I guess bulit-in lexers like HTML, CSS, JavaScript, etc. define their styles with large groups and a lot of keywords/operators etc.
get grouped together in such a way that excluding one style may lead to losing the style on some other stuff that I didn’t want to lose style.

I have the feeling that we have different understandings on the “excluded styles” feature.
This list is intended to be used only to avoid using your own regular expressions in certain situations.
Everything the lexers have done before is not affected.
For example, if we have the following Python code:

def my_function(p1, p2):
    ''' description was my_function(p1, p2) does '''

and we would have a regular expression to color p1 and p2, then it would be applied in the function definition line and in the comment line.
If this is desired, then we would not enter the comment styleID in the “excluded styles” list.
But if it is desired to color only the function definition line and not any mention in comments, then the comment styleIDs would be entered.

The script doesn’t change what the lexers do, that’s all still there.
The only thing it does is paint over the text of the regular expression matches.

Thanks for the discussion. Through this I realized that every regular expression should also have an “excluded styles” list to override the general one.
That means, if you don’t specify a list in a regular expression, the “global” list will be used, otherwise the list defined by this regular expression will be used.

[tyler durden]

@Ekopalypse
Thank you for clarifying this. I guess my understanding of the excluded styles functionality was a bit flawed, but everything makes more sense now.

Thanks for the discussion. Through this I realized that every regular expression should also have an “excluded styles” list to override the general one. That means, if you don’t specify a list in a regular expression, the “global” list will be used, otherwise the list defined by this regular expression will be used.

Glad I could help. If you do add more features, let me know! lol

Thank you again for all the help!

[Alan Kilborn]

@tyler-durden said in Using 'Enhance UDL Lexer' script to configure the syntax for multiple languages:

If you do add more features, let me know!

I guess you could just check back occasionally at its github page for any updates.

[tyler durden]

@Alan-Kilborn
You’re absolutely right. I didn’t mean it like “you must notify me”, it was more like “hey looking forward to seeing any updates” that’s why I added the “lol” at the end. I apologize if I didn’t come across well. Also, thank you the suggestion, I’ll definitely keep an eye on his GitHub page! :D