Community
    • Login

    Using 'Enhance UDL Lexer' script to configure the syntax for multiple languages

    Scheduled Pinned Locked Moved General Discussion
    19 Posts 3 Posters 1.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • EkopalypseE
      Ekopalypse
      last edited by

      Ohhh - this changed more than I thought it might do but good to see
      that you already started to improve it - well done.

      I just was about to post the link to the repo with the modified version

      Regarding the question about Npp object, this is the object
      PythonScript itself defines within its cpp code. It uses boost::python to
      create the python objects.

      I will have a look at your version and see if I can “borrow” some code :-)

      1 Reply Last reply Reply Quote 3
      • EkopalypseE
        Ekopalypse @tyler durden
        last edited by Ekopalypse

        @tyler-durden @Alan-Kilborn

        I never timed a compiled version with a non-compiled version but I
        assume that the difference isn’t that big if there is a difference at all.
        I’ll try to measure this.

        @tyler-durden

        With the newest PythonScript version 1.5.4 there is no need to use
        ctypes in order to get the language name anymore. You can use
        notepad.getLanguageName(notepad.getLangType())

        A performance issue might arise for this part of the code

        for key in list(regexes.keys()):
        	EnhanceUDLLexer(regexes[key],key).main()
        

        This will lead to multiple callbacks calls. Every new language will
        create another set of callbacks and all do get called even only one language is used at a time.
        Switch the regexes on a bufferactivated event.

        EkopalypseE 1 Reply Last reply Reply Quote 2
        • EkopalypseE
          Ekopalypse @Ekopalypse
          last edited by Ekopalypse

          @tyler-durden @Alan-Kilborn

          I did the test with PythonScript3 and a compiled (pyc) and a non-compiled version.

          The result is that both are more or less equally fast.
          I used the script itself to measure how long it takes to format the code in an update ui event.
          Result: 0.8 - 2.0 milliseconds on my machine.
          Sometimes the non-compiled version was faster, next time the pyc version was faster.

          1 Reply Last reply Reply Quote 2
          • tyler durdenT
            tyler durden
            last edited by

            @Ekopalypse
            I followed your instructions regarding the use of ctypes and the execution of multiple callbacks. I’ve also read your EnhanceAnyLexer.py and “borrowed” a few things (a_few_things == 99% lol). I know it’s a bit pointless to do this since you’ve already wrote a great tool to control multiple lexers, but whatever…

            Differences include:

            • I don’t check if the version is 1.5.4.0 since I know my version,
            • I don’t have a rgb(r, g, b) method – I do that outside the EnhanceUDLLexer class,
            • I don’t register lexers – your advice about “switching the regexes on a bufferactivated event” was very helpful and now I just switch dictionaries inside the style method (self.all_regexes[self.current_language]) based on notepad.getLanguageName(notepad.getLangType()),
            • I kept the configure method,
            • and some other minor changes.

            I guess the real difference is the input method for each lexer style since the set_up_list at the beginning gets all rgb’s, regexes, etc… Here’s the final version:

            # -*- coding: utf-8 -*-
            # needs Python Script version 1.5.4.0
            from Npp import (notepad,editor,editor1,editor2,NOTIFICATION,
            				SCINTILLANOTIFICATION,INDICATORSTYLE,INDICFLAG,INDICVALUE)
            
            set_up_list = [ \
            
            # --------------------- CONFIGURATION AREA --------------------- #
            
            #  "LANGUAGE"
            #  R, G, B, "REGEX", MATCH_GROUP
            #  ...
            #  ...
            #  R, G, B, "REGEX", MATCH_GROUP
            #  [excluded_style_0, excluded_style_1, ..., excluded_style_23]
            
            "Go_Lang", ## UDL
            189, 147, 249, r"\d\.\d+", 0,
            139, 233, 253, r"(\w+)\(", 1,
            [1, 4, 6, 7, 16, 17, 18, 19],
            
            "Py_Lang", ## UDL
            189, 147, 249, r"\d\.\d+", 0,
            120, 221, 84, r"(\w+)\(", 1,
            240, 143, 47, r"\@\w+", 0,
            198, 95, 107, r"\.(\w+)", 1,
            86, 182, 194,r"(\w+)\.", 1,
            195, 250, 195, r"^class(.\w*)", 1,
            [1, 2, 3, 4, 6, 7, 12, 16, 17, 18, 19],
            
            "C_Lang", ## UDL
            183, 252, 55, r"(\w+)\(", 1,
            255, 121, 198, r"\#\w+", 0,
            90, 100, 250, r"\b(stack|node)\b",0,
            [1, 3, 4, 6, 7, 12, 16, 17, 18, 19],
            
            "JavaScript", ## not an UDL
            120, 221, 84, r"\((.*?)\)", 0,
            90, 100, 250, r"\b(var|require)\b",0,
            [1, 2, 3, 4, 6, 7, 12, 16, 17, 18, 19],
            
            "HTML", ## not an UDL
            90, 100, 250, r"\b(href)\b",0,
            189, 147, 249, r"\d\.\d+", 0,
            120, 221, 84, r"(\w+)\(", 1,
            [0],
            
            "CSS", ## not an UDL
            120, 221, 84, r"(.*?)\{", 1,
            255, 166, 85, r"[{}]", 0,
            255, 84, 18, r"[:;]", 0,
            [0],
            
            "R_Lang", ## UDL
            120, 221, 84, r"(\w+)\(", 1,
            [1, 3, 4, 6, 7, 12, 16, 17, 18, 19], ## add comma at the end
            
            # --------------------- CONFIGURATION AREA --------------------- #
            
            None
            ]#end of set_up_list's square brackets
            
            def language_set_up(set_up_list):
            	all_regexes = {}
            	excluded_styles = {}
            	regexes = {}
            	pattern = lambda x,k: 5*x + k
            	change_rgb = lambda r,g,b: (b << 16) + (g << 8) + r
            
            	row_count = 0
            
            	while True:
            		if isinstance(set_up_list[pattern(row_count,1)], list):
            			LANGUAGE = set_up_list[0]
            			all_regexes[LANGUAGE] = regexes
            			excluded_styles[LANGUAGE] = set_up_list[pattern(row_count,1)]
            			if set_up_list[pattern(row_count,2)] != None:
            				del set_up_list[0:pattern(row_count,2)]
            				regexes = {}
            				row_count = 0
            			else:
            				break
            
            		ID = row_count
            		R = set_up_list[pattern(row_count,1)]
            		G = set_up_list[pattern(row_count,2)]
            		B = set_up_list[pattern(row_count,3)]
            		REGEX = set_up_list[pattern(row_count,4)]
            		MATCH_GROUP = set_up_list[pattern(row_count,5)]
            		
            		regexes[(ID, change_rgb(R,G,B) | INDICVALUE.BIT)] = (REGEX,MATCH_GROUP)
            		
            		row_count += 1
            
            	all_regexes[LANGUAGE] = regexes	
            	return all_regexes, excluded_styles
            
            regexes, excluded_styles = language_set_up(set_up_list)
            
            class EnhanceUDLLexer:
            	
            	def __init__(self, all_regexes, excluded_styles):
            		self.INDICATOR_ID = 0
            		self.doc_is_of_interest = False
            		self.all_regexes = all_regexes
            		self.excluded_styles = excluded_styles
            		self.current_language = ""
            		self.configure()
            
            	def configure(self):
            		editor1.indicSetStyle(self.INDICATOR_ID, INDICATORSTYLE.TEXTFORE)
            		editor1.indicSetFlags(self.INDICATOR_ID, INDICFLAG.VALUEFORE)
            		editor2.indicSetStyle(self.INDICATOR_ID, INDICATORSTYLE.TEXTFORE)
            		editor2.indicSetFlags(self.INDICATOR_ID, INDICFLAG.VALUEFORE)
            		editor.callbackSync(self.on_updateui, [SCINTILLANOTIFICATION.UPDATEUI])
            		editor.callbackSync(self.on_marginclick, [SCINTILLANOTIFICATION.MARGINCLICK])
            		notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED])
            		notepad.callback(self.on_bufferactivated, [NOTIFICATION.BUFFERACTIVATED])
            
            	def paint_it(self, color, match_position, length, start_position, end_position):
            		if (match_position + length < start_position or
            			match_position > end_position or
            			editor.getStyleAt(match_position) in self.excluded_styles[self.current_language]):
            			return
            
            		editor.setIndicatorCurrent(0)
            		editor.setIndicatorValue(color)
            		editor.indicatorFillRange(match_position, length)
            
            
            	def style(self):
            		start_line = editor.docLineFromVisible(editor.getFirstVisibleLine())
            		end_line = editor.docLineFromVisible(start_line + editor.linesOnScreen())
            		if editor.getWrapMode():
            			end_line = sum([editor.wrapCount(x) for x in range(end_line)])
            
            		onscreen_start_position = editor.positionFromLine(start_line)
            		onscreen_end_pos = editor.getLineEndPosition(end_line)
            
            		editor.setIndicatorCurrent(0)
            		editor.indicatorClearRange(0, editor.getTextLength())
            		for color, regex in self.all_regexes[self.current_language].items():
            			editor.research(regex[0],
            							lambda m: self.paint_it(color[1],
            													m.span(regex[1])[0],
            													m.span(regex[1])[1] - m.span(regex[1])[0],
            													onscreen_start_position,
            													onscreen_end_pos),
            							0,
            							onscreen_start_position,
            							onscreen_end_pos)
            
            	def check_lexer(self):
            		current_language = notepad.getLanguageName(notepad.getLangType()).replace('udf - ','')
            		if current_language in self.all_regexes:
            			self.doc_is_of_interest = True
            			self.current_language = current_language
            
            	def on_marginclick(self, args):
            		if args['margin'] == 2 and self.doc_is_of_interest:
            			self.style()
            
            	def on_bufferactivated(self, args):
            		self.check_lexer()
            
            	def on_updateui(self, args):
            		if self.doc_is_of_interest:
            			self.style()
            
            	def on_langchanged(self, args):
            		self.check_lexer()
            
            
            	def main(self):
            		self.on_bufferactivated(None)
            		self.on_updateui(None)
            
            
            Enhance = EnhanceUDLLexer(regexes,excluded_styles)
            Enhance.main()
            

            I timed this script, my old script (with multiple callbacks) and your EnhanceAnyLexer.py script. All of them were timed using the same method and with the same number of lexers (7 = 4 UDLs and 3 built-in lexers). I did not change the registration process when timing EnhanceAnyLexer.py. In other words, I added styles as expected, for example: r_regexes = _dict() => r_regexes[(0, (120, 221, 84))] = (r"(\w+)\(", 1) => r_excluded_styles = [1, 3, 4, 6, 7, 12, 16, 17, 18, 19].

            I timed all scripts by adding a the following to them:

            import time
            start = time.time()
            #
            # actual lexer script
            #
            end = time.time()
            print "Lexer Time: %.2f ms" % (1000*(end-start))
            

            I don’t know if this method is accurate or appropriate for this particular situation (if you know of a better way, please let me know), but both EnhanceAnyLexer.py and my new script got an average of 30 ms, which seems to indicate that I don’t have any performance issues like multiple callbacks, etc. However, my old script, just as you said, got an average of 3250 ms lol … 7 lexers with multiple callbacks do add up quickly.

            So, that’s it. I’m pretty happy with the result. Two things I’d like to ask are: [1] is it possible to control bold/italic/underline/other rich text stuff via Python? [2] instead of excluding entire styles, is it possible to excluded specific keywords/operators/delimiters/etc. like excluding only the word def for example.?

            Thank you!!

            EkopalypseE 1 Reply Last reply Reply Quote 2
            • EkopalypseE
              Ekopalypse @tyler durden
              last edited by Ekopalypse

              @tyler-durden

              I know it’s a bit pointless to do this …

              I even think this makes sense if you want to extend/modify existing code.
              Personally I learn fastest this way. Good job.

              Measuring execution time is one of those things.
              Actually you should make sure that only Npp and PythonScript(PS) are used
              to make sure that other plugins don’t interfere.
              Because of the accuracy I think that tests that take several milliseconds are relatively accurate,
              unless you are programming for a RealTime system, but then Python is probably the wrong language as well.
              Higher accuracy could be achieved with ctypes and highperformance counter,
              but do we shoot sparrows with cannons here?
              But if the question was about how to code timing functions, I prefer decorators.
              I have a user startup.py file which is automatically executed on every startup, because it is called startup.py.
              Please do not confuse it with the delivered startup.py from PS. If you make changes there, they would be lost when you update.
              In this user startup.py I have several functions like

              from time import perf_counter
              def timeit(func):
                  def wrapped(*args, **kwargs):
                      s = perf_counter()
                      x = func(*args, **kwargs)
                      e = perf_counter()
                      print(f'Calling {func.__name__} took {e-s}')
                      return x
                  return wrapped
              

              for Python2 it must be changed to

              from time import time
              def timeit(func):
                  def wrapped(*args, **kwargs):
                      s = time()
                      x = func(*args, **kwargs)
                      e = time()
                      print('Calling {} took {}'.format(func.__name__, e-s))
                      return x
                  return wrapped
              

              Then all you need to do is

              @timeit
              def my_function_to_test(p1,p2):
                  pass
              

              I’m pretty happy with the result.

              And that counts, script is readable, extendable … what would one else.

              1. Yes and no. Scintilla knows two ways to colorize text, Styles and Indicators.
                Styles offer you the full range of possibilities scintilla offers to manipulate text attributes.
                Indicators can only change the font color. In general both can be manipulated from PS.
                The reason why indicators are used here is because Lexer uses styles.
                If we would also use styles we would always be in “clinch” with the lexers, whose definition is the final one.

              2. Currently the script doesn’t give that, maybe use a regular expression that takes that into account?
                Or am I in the middle of something and don’t understand what you mean?

              1 Reply Last reply Reply Quote 3
              • tyler durdenT
                tyler durden
                last edited by

                @Ekopalypse
                First of all, thank you for the info about measuring execution time.

                My lexer script is saved here: C:\Users\rtl\AppData\Roaming\Notepad++\plugins\config\PythonScript\scripts\EnhanceUDLLexer.py and to automatically execute it on every startup I added the following line import EnhanceUDLLexer at the end of the delivered startup.py from PythonScript (which is located here C:\Program Files (x86)\Notepad++\plugins\PythonScript\scripts\startup.py).

                I imported the lexer script (EnhanceUDLLexer.py) from the delivered startup.py because I didn’t know you could have other startup.py scripts. Which path should I choose for my user startup file to make it run on every startup? Is it bad to alter the delivered startup.py like I did with the extra import?

                About my previous questions:
                [1] I think I understand the basics of this issues. That’s not a problem though…I asked just out of curiosity.

                [2] I guess bulit-in lexers like HTML, CSS, JavaScript, etc. define their styles with large groups and a lot of keywords/operators etc. get grouped together in such a way that excluding one style may lead to losing the style on some other stuff that I didn’t want to lose style. But…now that I think about it, it might be easier to just copy the built-in stuff into an UDL and make all the changes I want. Never mind about this question.

                Thanks again!

                EkopalypseE 1 Reply Last reply Reply Quote 1
                • EkopalypseE
                  Ekopalypse @tyler durden
                  last edited by

                  @tyler-durden

                  Which path should I choose for my user startup file to make it run on every startup?

                  Just create a new script and name it startup.py.
                  It will be automatically put into your user script directory.

                  Is it bad to alter the delivered startup.py like I did with the extra import?

                  The included startup.py will be replaced when there is a PythonScript update, otherwise, no.

                  I guess bulit-in lexers like HTML, CSS, JavaScript, etc. define their styles with large groups and a lot of keywords/operators etc.
                  get grouped together in such a way that excluding one style may lead to losing the style on some other stuff that I didn’t want to lose style.

                  I have the feeling that we have different understandings on the “excluded styles” feature.
                  This list is intended to be used only to avoid using your own regular expressions in certain situations.
                  Everything the lexers have done before is not affected.
                  For example, if we have the following Python code:

                  def my_function(p1, p2):
                      ''' description was my_function(p1, p2) does '''
                  

                  and we would have a regular expression to color p1 and p2, then it would be applied in the function definition line and in the comment line.
                  If this is desired, then we would not enter the comment styleID in the “excluded styles” list.
                  But if it is desired to color only the function definition line and not any mention in comments, then the comment styleIDs would be entered.

                  The script doesn’t change what the lexers do, that’s all still there.
                  The only thing it does is paint over the text of the regular expression matches.

                  Thanks for the discussion. Through this I realized that every regular expression should also have an “excluded styles” list to override the general one.
                  That means, if you don’t specify a list in a regular expression, the “global” list will be used, otherwise the list defined by this regular expression will be used.

                  1 Reply Last reply Reply Quote 3
                  • tyler durdenT
                    tyler durden
                    last edited by

                    @Ekopalypse
                    Thank you for clarifying this. I guess my understanding of the excluded styles functionality was a bit flawed, but everything makes more sense now.

                    Thanks for the discussion. Through this I realized that every regular expression should also have an “excluded styles” list to override the general one. That means, if you don’t specify a list in a regular expression, the “global” list will be used, otherwise the list defined by this regular expression will be used.

                    Glad I could help. If you do add more features, let me know! lol

                    Thank you again for all the help!

                    Alan KilbornA 1 Reply Last reply Reply Quote 1
                    • Alan KilbornA
                      Alan Kilborn @tyler durden
                      last edited by

                      @tyler-durden said in Using 'Enhance UDL Lexer' script to configure the syntax for multiple languages:

                      If you do add more features, let me know!

                      I guess you could just check back occasionally at its github page for any updates.

                      1 Reply Last reply Reply Quote 1
                      • tyler durdenT
                        tyler durden
                        last edited by

                        @Alan-Kilborn
                        You’re absolutely right. I didn’t mean it like “you must notify me”, it was more like “hey looking forward to seeing any updates” that’s why I added the “lol” at the end. I apologize if I didn’t come across well. Also, thank you the suggestion, I’ll definitely keep an eye on his GitHub page! :D

                        1 Reply Last reply Reply Quote 2
                        • First post
                          Last post
                        The Community of users of the Notepad++ text editor.
                        Powered by NodeBB | Contributors