Assign Language by Line
-
-
after thinking about it for some time I can imagine how this could be achieved.
In theory:- using a hidden scintilla document
- and a script for selecting which lexer should be used
workflow:
- select text
- run script and select lexer from a messagebox, prompt, or whatever ui
- now, under the hood, the script would get the text selected and paste it into the
hidden scintilla document and sets the lexer. Then it reads the styles assigned to
the text and sets it in the real document
Is this possible? I don’t know, I never done something like this but is something I’m eager to find out. So I need some time playing around and let you know.
Concerning the “Can I stop npp, restart some time later and the coloring is still there?”,
I would say, yes, if a strict policy is in place. Something like a file must be saved always and this file is not allowed to be manipulated outside from npp. -
I’ve tested the concept and it looks like it could work, BUT
there are some strange things happening.Assuming one wants to color every word of a text in red.
When using these three lines to do the coloringeditor.setIndicatorCurrent(INDICATOR_ID) editor.setIndicatorValue(color | SC_INDICVALUEBIT) editor.indicatorFillRange(start, length)there is an extreme difference in performance if you call it directly or via a callback
like SCINTILLANOTIFICATION.UPDATEUI.If one is using a callback it takes 11ms to color 377 words,
if calling it directly, it takes 6 seconds. ???In addition, if calling it directly, it happens that not all words get colored.
Sometimes it misses 2 or 3 or … words even so the debug shows that those missed
words are actually processed. ???What haven’t I understood?
And some more strange things. Just for the fun, I’ve tested a python only version,
means building a window and embedding scintilla in pure python,
but python3 I must admit, it took 3ms to do the same thing.
What is slowing down when using PS?? -
I think there are some “slowness” gremlins in Pythonscript that no one has totally figured out yet. Here’s a reference to another type, maybe some hints here for you Eko because from observation it definitely seems like you have the skills to maybe solve it or at least root cause it. I was hoping @Claudia-Frank would solve these things, but no luck and she has disappeared. Maybe you are the new Claudia. ;)
I’ve tested a python only version, means building a window and embedding scintilla in pure python
I would like to understand how to do that, sure, just for fun. :)
-
Alan, thanks for the link but, oh lord, if they were confused how should I find it?
They have far more experience in c++ than I do. Phew.Maybe you are the new Claudia.
But I don’t have to turn myself into a woman, or? LOL :-D
I would like to understand how to do that, sure, just for fun. :)
:-) Claudia posted this and I slightly modified it.
def MainLoop(self): self.ColorEachWordRed() gui.PumpMessages() def ColorEachWordRed(self): SC_INDICVALUEBIT = 0x1000000 INDICATOR_ID = 8 TEXTFORE = 17 VALUEFORE = 1 # editor1.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) self.scintilla_direct_function(self.sci_direct_pointer, 2080, INDICATOR_ID, TEXTFORE) # editor1.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) self.scintilla_direct_function(self.sci_direct_pointer, 2684, INDICATOR_ID, VALUEFORE) t1_start = time.time() i = 0 for match in re.finditer(b'\w+', self.text): i += 1 start, end = match.span() self.scintilla_direct_function(self.sci_direct_pointer, 2500, INDICATOR_ID) self.scintilla_direct_function(self.sci_direct_pointer, 2502, 255 | SC_INDICVALUEBIT) self.scintilla_direct_function(self.sci_direct_pointer, 2504, start, end-start) t1_stop = time.time() print('Colored %d items' % i) print("Elapsed time: %.6f" % (t1_stop-t1_start))self.text, my test data, is this
self.text = b'''Hello World here is python Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed molestie nisl placerat, ultricies risus nec, aliquam nunc. Nunc urna metus, molestie ut augue vitae, posuere faucibus elit. Donec sodales est tortor, eu aliquet massa hendrerit et. Aliquam pretium fermentum volutpat. Aliquam et lorem id orci tincidunt porta. Duis vehicula, mi eu tempor cursus, mauris ex efficitur libero, ac ultricies dui enim eu dui. Proin vel diam pulvinar, consequat libero ac, interdum lorem. Nam interdum ut sapien eu viverra. Nunc molestie efficitur sollicitudin. Morbi vel fermentum velit. 123 Aliquam nec lectus a enim vulputate pretium. Nunc elementum tristique porttitor. Curabitur malesuada sed augue sed lobortis. Vivamus sit amet pharetra dolor. Etiam finibus gravida felis, sed porttitor sapien. In lobortis purus eu ipsum auctor varius. Cras eu tempor urna, quis iaculis nunc. Ut purus lectus, ultrices congue elementum quis, pellentesque vel eros. Mauris eget dolor non dolor finibus mollis vel et elit. Quisque vestibulum orci non nulla pulvinar, ac efficitur velit tempor. Cras non sem pellentesque, feugiat sapien sed, commodo elit. Cras dolor augue, eleifend vel urna vitae, consequat blandit lectus. Curabitur porttitor neque in mi ornare dictum. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Duis ante nisl, facilisis tincidunt odio quis, placerat molestie dolor. 456 Phasellus rutrum posuere convallis. Proin dictum ex eget nisl eleifend tempus. Cras id lorem fermentum, molestie quam eu, commodo odio. Curabitur id quam tellus. Suspendisse potenti. Etiam egestas tincidunt dui, quis ornare nulla rhoncus quis. Suspendisse sodales elit vel felis vestibulum mollis. 789 Aenean quis massa laoreet, sagittis est ac, bibendum risus. Praesent eu sapien faucibus, porta orci non, ultricies elit. Morbi a dapibus neque. Nunc at odio eget risus egestas efficitur. Duis ullamcorper velit diam, et volutpat lacus malesuada eget. Mauris dignissim tincidunt rutrum. Sed molestie ut nisl ac dictum. Vestibulum a libero ut turpis tristique vehicula. Nam lobortis, leo vitae pellentesque sollicitudin, dolor leo viverra enim, sit amet ullamcorper metus magna eget ipsum. Ut ac eros porta, lacinia sem ac, posuere elit. Nam at efficitur orci, nec volutpat ligula. Quisque non nisi sit amet arcu accumsan viverra quis et tellus. Fusce accumsan, nunc vel semper sollicitudin, elit ante congue velit, vitae dignissim tellus risus ac justo. Duis non leo lorem. 000 '''Make sure your SciLexer.dll is in the same directory as your script and, of course,
that python bitness matches SciLexer.dll bitness.Edit: You need to pip install win32 package.
-
just in case you want to test the other two
direct call
from Npp import editor, notepad, INDICATORSTYLE, INDICFLAG import time SC_INDICVALUEBIT = 0x1000000 INDICATOR_ID = 8 editor1.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor1.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) editor2.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor2.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) def paint_it(color, start, length): editor.setIndicatorCurrent(INDICATOR_ID) editor.setIndicatorValue(color | SC_INDICVALUEBIT) editor.indicatorFillRange(start, length) s1 = time.time() matches = [] editor.research(r'\w+', lambda m: matches.append(m.span())) for match in matches: paint_it(255, match[0], match[1] - match[0]) print time.time() - s1via callback
from Npp import editor, notepad, INDICATORSTYLE, INDICFLAG import time SC_INDICVALUEBIT = 0x1000000 INDICATOR_ID = 8 editor1.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor1.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) editor2.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor2.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) def paint_it(color, start, length): editor.setIndicatorCurrent(INDICATOR_ID) editor.setIndicatorValue(color | SC_INDICVALUEBIT) editor.indicatorFillRange(start, length) def on_updateui(args): s1 = time.time() matches = [] editor.research(r'\w+', lambda m: matches.append(m.span())) for match in matches: paint_it(255, match[0], match[1] - match[0]) print time.time() - s1 editor.clearCallbacks([SCINTILLANOTIFICATION.UPDATEUI]) editor.callbackSync(on_updateui, [SCINTILLANOTIFICATION.UPDATEUI])Edit: be careful about the callback as it will color every document, use it, maybe, in its own npp instance :-)
-
@Ekopalypse said:
Claudia posted this and I slightly modified it.
Ah…well it appears that even if you are not the new Claudia, you are at least “channeling” the old Claudia. That posting from Claudia was only 4 days old…seems she is still in the Scintilla game even if not the N++ game…maybe she is making the next new great competitor for N++??
if they were confused how should I find it?
You seem to have a knack for this stuff (way beyond my basic skill); maybe all it takes is a pair of fresh eyes?
BTW, I have a new (perhaps tough?) problem for Pythonscript that I’ll be posting here (well, in a brand-new thread) soon…hopefully you’ll be watching for it. :)
-
she recently posted at pythonscript github so I’m hoping that she might find her
way back home and of course, I’m always looking for python related stuff :-D -
Just in case you wondered what happend, well, just real life or better wife :-)
I guess I have something you can play with.
The save option, meaning remembering what was colored where and how, needs to be discussed.I see three option
- use an additional file with the same name as the current file but with a different unique extension, which contains colors and position
- use one file which contains the names of the files which were used and there colors and position
- not using any file at all, but a marker in the original file which indicates where a different languages starts and stops.
Concerning the script itself.
You have to install python script plugin and, for the time being, avoid updating to npp 7.7 or newer unless PythonScript gets updated to reflect the changes made in npp7.7
Once PS is installed you have to call it for every document, which by the way needs to be set to normal text, in which you want to see different languages colored once. Then select the text and use the language menu to chose the lexer of your choice. Done.Two remarks.
UDLs are currently supported by a hack. When using an UDL the script opens a new tab, pastes the text, sets the udl lexer, gets the colors and closes the tab, whereas if using a builtin lexer a hidden scintilla component is used.Only foreground colors are supported, meaning no font settings like bold or italic. That is because indicators do not support these.
Using indicators instead of styles has the advantage that switching
between documents is possible without loosing the styling information.The script, which I will post in the next post, itself contains TESTDATA.
If you want to use it, copy everything from line 291 to 360 and paste it into a new tab.Is it bug free? Most likely not.
In case your finding something weird or in case it doesn’t do what you want it to do, let me know. -
# -*- coding: utf-8 -*- from Npp import editor, notepad, INDICATORSTYLE, INDICFLAG, LANGTYPE, NOTIFICATION import xml.etree.ElementTree as et import os import json from pprint import pformat try: TextStyler().register_document() except NameError: SC_INDICVALUEBIT = 0x1000000 INDICATOR_ID = 8 class SingletonTextStyler(type): ''' Ensures, more or less, that only one instance of the main class can be instantiated ''' _instance = None def __call__(cls, *args, **kwargs): if cls._instance is None: cls._instance = super(SingletonTextStyler, cls).__call__(*args, **kwargs) return cls._instance class TextStyler(object): __metaclass__ = SingletonTextStyler def __init__(self): self.hidden_scintilla = notepad.createScintilla() editor1.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor1.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) editor2.indicSetStyle(INDICATOR_ID, INDICATORSTYLE.TEXTFORE) editor2.indicSetFlags(INDICATOR_ID, INDICFLAG.VALUEFORE) self.npp_config_dir = notepad.getNppDir() self.styler_dict = dict() self.active_theme = et.parse(self.get_active_styler_xml()) # self.udls = self.create_udls_xml() self.lexer_keywords = self.get_langs_xml() self.list_of_document_ids = [] self.current_lexer = 'null' self.default_id = 0 self.memory_file = os.path.join(notepad.getPluginConfigDir(), 'TextStylerMemory.txt') self.remember_styles = None notepad.callback(self.on_langchanged, [NOTIFICATION.LANGCHANGED]) notepad.callback(self.on_fileclosed, [NOTIFICATION.FILECLOSED]) notepad.callback(self.on_filesaved, [NOTIFICATION.FILESAVED]) notepad.callback(self.on_shutdown, [NOTIFICATION.SHUTDOWN]) # there is an peformance issue when calling paint_it directly, # therefore we are hacking around langchanged notification def on_langchanged(self, args): ''' Gets called by npp when language change is deteced Resets to null lexer and calls main method ''' if (args['bufferID'] in self.list_of_document_ids): if notepad.getLangType() == LANGTYPE.TXT: if self.current_lexer != 'null': self.main(self.current_lexer) else: notepad.messageBox('Expected a lexer but null found', 'Error',0) else: self.current_lexer = editor.getLexerLanguage() if self.current_lexer == 'user': self.current_lexer = notepad.getLanguageName(notepad.getLangType()) notepad.setLangType(LANGTYPE.TXT) def on_shutdown(self, args): ''' Gets called by npp when shutting down TODO: store knwon files and their colors which then would be reused on fileopen/bufferactived callback ''' notepad.destroyScintilla(self.hidden_scintilla) def on_filesaved(self, args): ''' Gets called by npp when a buffer is saved TODO: check if doc is of interest if so, get current filename check if doc is empty if so delete from memory if not store current colors and their position ''' if (args['bufferID'] in self.list_of_document_ids): current_file = notepad.getCurrentFilename() from pprint import pformat values = [editor.indicatorValueAt(INDICATOR_ID,x) for x in range(editor.getTextLength())] prev_value = 0 fill_range = 1 sorted_values = [] for offset, value in enumerate(values): if value == 0: continue if prev_value == value: fill_range += 1 else: sorted_values.append((value, offset-fill_range, fill_range)) prev_value = value fill_range = 1 print(pformat(sorted_values)) def on_fileclosed(self, args): ''' Gets called by npp when a buffer gets closed Removes unneeded buffer ids from internal list ''' if (args['bufferID'] in self.list_of_document_ids): self.list_of_document_ids.remove(args['bufferID']) def get_active_styler_xml(self): ''' returns the xml path of the current theme ''' xml_file = os.path.join(self.npp_config_dir, 'config.xml') xml_doc = et.parse(xml_file) return xml_doc.find('GUIConfigs/GUIConfig[@name="stylerTheme"]').get('path') # def create_udls_xml(self): # ''' merge all udls from userDefineLangs into one big xml ''' # udls_dir = os.path.join(self.npp_config_dir, 'userDefineLangs') # udl_files = os.listdir(udls_dir) # root = et.Element('NotepadPlus') # for udl_file in udl_files: # xml_doc = et.parse(os.path.join(udls_dir, udl_file)) # user_lang = xml_doc.find('UserLang') # root.append(user_lang) # return root def get_lexer_styles(self, lexername): ''' Creates the styling dictionary by reading the styler xml ''' if lexername.startswith('udf - '): return # as long as hidden scintilla doesn't do the styling we are done here # tag = 'UserLang[@name="%s"]/Styles/WordsStyle' % lexername[6:] # lexer_styles = self.udls.findall(tag) # self.default_id = 0 # for id, result in enumerate(lexer_styles): # fgColor = result.attrib.get('fgColor', None) # if not fgColor: # notepad.messageBox('Received unexpected value\nid: {}\nfgColor: {}'.format(id, fgColor), 'Error',0) # return # red, green, blue = bytearray.fromhex(fgColor) # color = (blue<<16) + (green<<8) + red # self.styler_dict[int(id)] = color tag = 'LexerStyles/LexerType[@name="%s"]/WordsStyle' % lexername lexer_styles = self.active_theme.findall(tag) for result in lexer_styles: if result.attrib.get('name', None) == 'DEFAULT': id = result.attrib.get('styleID', None) self.default_id = int(id) else: id = result.attrib.get('styleID', None) fgColor = result.attrib.get('fgColor', None) if not fgColor: notepad.messageBox('Received unexpected value\nid: {}\nfgColor: {}'.format(id, fgColor), 'Error',0) return red, green, blue = bytearray.fromhex(fgColor) color = (blue<<16) + (green<<8) + red self.styler_dict[int(id)] = color def get_langs_xml(self): ''' returns a dictionary which contains all keywords for all defined builtin languages ''' xml_file = os.path.join(self.npp_config_dir, 'langs.xml') xml_doc = et.parse(xml_file) lexer_keywords = dict() for lang in xml_doc.findall('Languages/Language'): lexer_keywords[lang.attrib['name']] = lang.findall('Keywords') return lexer_keywords def paint_it(self, color, start, length): ''' Does the coloring by using an indicator ''' editor.setIndicatorCurrent(INDICATOR_ID) editor.setIndicatorValue(color | SC_INDICVALUEBIT) editor.indicatorFillRange(start, length) def main(self, lexer): ''' Gathers the needed information for the current selected lexer and sets these in the hidden scintilla. Calls hidden scintillas colourise method and retrieves the needed colors and their position. Finally sets it in the active document ''' text = editor.getSelText() text_length = len(text) if not text_length: notepad.messageBox('Nothing selected!', 'Error',0) return if not lexer: notepad.messageBox('lexer expected, received: {}'.format(lexer), 'Error',0) return offset = min(editor.getSelectionStart(), editor.getSelectionEnd()) # unfortunately udls behave different than builtin lexers # there for this hack until the secrets get revealed if lexer.startswith('udf - '): active_buffer = notepad.getCurrentBufferID() notepad.new() editor.setText(text) notepad.runMenuCommand('Language', lexer[6:]) styles = [(pos, editor.styleGetFore(editor.getStyleAt(pos))) for pos in range(text_length)] editor.undo() notepad.close() notepad.activateBufferID(active_buffer) for position, color in styles: self.paint_it(color[0] + (color[1] << 8) + (color[2] << 16), offset+position, 1) return # we are done here # else part - aka builtin lexer self.get_lexer_styles(lexer) keywords = self.lexer_keywords.get(lexer, None) self.hidden_scintilla.styleClearAll() self.hidden_scintilla.setLexerLanguage(lexer) if keywords: # not all lexers do have keywords, e.g. xml lexer for i, _keywords in enumerate(keywords): self.hidden_scintilla.setKeyWords(i, _keywords.text) self.hidden_scintilla.setText(text) self.hidden_scintilla.colourise(0, text_length) styles = [(pos, self.hidden_scintilla.getStyleAt(pos)) for pos in range(text_length)] color = self.styler_dict.get(self.default_id, None) if color is None: return self.paint_it(color, offset, text_length) # TODO: can default color different previous_style = [] sorted_styles = dict() for pos, style in styles: if previous_style: if previous_style[0] == style: previous_style[2] += 1 else: if previous_style[0] != 0: if previous_style[0] in sorted_styles: sorted_styles[previous_style[0]].append((previous_style[1],previous_style[2])) else: sorted_styles[previous_style[0]] = [(previous_style[1],previous_style[2])] previous_style = [style, pos, 1] else: previous_style = [style, pos, 1] for k, v in sorted_styles.items(): color = self.styler_dict.get(k, None) if not color: continue for _v in v: self.paint_it(color, offset+_v[0], _v[1]) def register_document(self): ''' store current buffer id in internal list ''' id = notepad.getCurrentBufferID() if id not in self.list_of_document_ids: self.list_of_document_ids.append(id) TextStyler().register_document() print('done') # -------------------------------------------------------------------------- # TESTDATA TESTDATA = ''' Start with an perl example use strict; # comment use warnings; print "Hello, World!\n"; ------------------------------------------------------------------------------ here we have an cpp example #include <iostream> using namespace std; // line comment int main() { /* multi line comment */ cout << "Hello, World!"; return 0; } ------------------------------------------------------------------------------ followed by an rust example fn main() { // Print text to the console println!("Hello World!"); } ------------------------------------------------------------------------------ an xml example <tags> <tag id="0" value="text" /> </tags> ------------------------------------------------------------------------------ an python example class test(object): def __init__(self): self.greet = 'Hello World!' # comment def do_greet(self): print(self.greet) ------------------------------------------------------------------------------ and finally udl markdown \1-2-3 *multiple italic*, **multi bold** and ***multi bold italic*** I have *a pen **I have an apple** uuh* Apple pen. I have **a pen *I have a pineapple* uuh** Pineapple pen! I have *a pen __I have an apple__ uuh* Apple pen. I have **a pen _I have a pineapple_ uuh** Pineapple pen! \* apple \* pen \* <!-- escape test \* t \* t \* t --> ```js var a = 1 * 1 ``` ****** <!-- hr --> ## Bullet point test - normal text * This should be normal text * mess up *some thing* (\* suppose \* it's \* normal) * This should be normal text ''' # --------------------------------------------------------------------------
