[vi simulator] how to highlight a word

Scott Sumner

The Pythonscript docs say that I can use editor.callbackSync() to make it synchronous, but then it goes on to say that if I do that, I can’t call editor.findText(), which this script uses. :(

Claudia Frank

Hello Scott,

you are right, using asynchronous callback leads to the update flickering.
I’m not quite sure why this happens.
I’m using the synchronous callback together with the research function and this
seems to work well. Your code would look like

INDICATOR_TO_USE = 12
editor1.indicSetStyle(INDICATOR_TO_USE, INDICATORSTYLE.ROUNDBOX)
editor2.indicSetStyle(INDICATOR_TO_USE, INDICATORSTYLE.ROUNDBOX)
editor1.indicSetAlpha(INDICATOR_TO_USE, 55)
editor2.indicSetAlpha(INDICATOR_TO_USE, 55)
editor1.indicSetOutlineAlpha(INDICATOR_TO_USE, 255)
editor2.indicSetOutlineAlpha(INDICATOR_TO_USE, 255)

import re

def callback_sci_UPDATEUI(args):
    print 'callback_sci_UPDATEUI'
    
    def match_found(m):
        editor.setIndicatorCurrent(INDICATOR_TO_USE)
        editor.indicatorFillRange(m.span(0)[0], m.span(0)[1] - m.span(0)[0])
    
    def getRangeOnScreen():
        print 'getRangeOnScreen'
        firstLine = editor.getFirstVisibleLine()
        lastLine = firstLine + editor.linesOnScreen()
        startPos = editor.positionFromLine(firstLine)
        endPos = editor.getLineEndPosition(lastLine)
        return (startPos, endPos)

    def clearIndicatorOnScreen():
        print 'clearIndicatorOnScreen'
        (s, e) = getRangeOnScreen()
        editor.indicatorClearRange(s, e - s)

    editor.setIndicatorCurrent(INDICATOR_TO_USE)

    if not editor.getSelectionEmpty():
        clearIndicatorOnScreen()
        return False
    else:
        startWord = editor.wordStartPosition(editor.getCurrentPos(), True)
        endWord = editor.wordEndPosition(startWord, True)

        if startWord == endWord:
            clearIndicatorOnScreen()
            return False
        else:
            word = editor.getTextRange(startWord, endWord)
            print 'word:{}'.format(word)
            clearIndicatorOnScreen()

            (startPos, endPos) = getRangeOnScreen()
            # temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, startPos, endPos, word)
            editor.research(word, match_found, re.IGNORECASE) 
            # while temp != None:
                # (s, e) = temp
                # editor.indicatorFillRange(s, e - s)
                # temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, e, endPos, word)

editor.callbackSync(callback_sci_UPDATEUI, [SCINTILLANOTIFICATION.UPDATEUI])

I know you could have done this yourself but thought …

Cheers
Claudia

dail

I’m not familiar enough with the internal Scintilla code but I think it is making sure it doesn’t get stuck in one of these types of loops. I added a bit of extra code to my plugin to log exactly when notifications are getting received. It shows when it enters and leaves the notifications. As shown below, it enters the SCN_UPDATEUI, at which point it calls my Lua callback which adds 3 indicators, thus the 3 SCN_MODIFIED pairs, then leaves the SCN_UPDATEUI notification.

->SCN_UPDATEUI
->SCN_MODIFIED
<-SCN_MODIFIED
->SCN_MODIFIED
<-SCN_MODIFIED
->SCN_MODIFIED
<-SCN_MODIFIED
<-SCN_UPDATEUI

I would say between this and the fact that @Claudia-Frank successfully used the synchronous callbacks, means that your code was receiving these notifications out of order due to the asynchronous callbacks (which is a problem I ran into with the PythonScript a while ago).

Scott Sumner

Claudia and dail,

Thanks for the follow-up on this. This started out as just a curiosity to me, but in the end I learned a lot and I liked the functionality of this little script so much that I integrated it into my startup.py so that it is a permanent part of my Notepad++. I even tweaked it so that the highlight at the caret uses a second indicator to give it more emphasis than the other matches within the viewing window. Thanks again.

Scott Sumner

After using what I came up with for a while, I happened to notice that when the current document also exists as clone (that is, is in both views), when the caret is on a word the UPDATEUI callback fires continually. The intended highlighting works, but with the callback firing continually, this is not good. I suspected this when the editing tabs in both views disappeared. I put in some “I’m here” output to the pythonscript console window at the top of the callback, and suspicions were confirmed.

What I’m unsure of is how to fix it. I went back to @Claudia-Frank 's original script above and made sure the same problem exists there and that it wasn’t something I introduced. Claudia or @dail , any pointers in the right direction for a fix would be appreciated.

dail

@Scott-Sumner

I had never noticed that before (mostly because I rarely clone documents) but I am seeing the same thing when using Lua. I already tried a few things to fix it but did not have any luck.

Claudia Frank

@Scott-Sumner

Hi Scott,

not at home, so just a quick info.
Afaik, cloned documents are just two references to the same object
therefore your experience can be explained. What I would try to do
is to see if it can be made unique again like checking if
file, which triggers the coloring, is different from the cloned one.
Like current document call, it gives back bufferid, view and file position.
Maybe create an identifier out of it and check this before callback gets executed.

I will give it a try, once I’m at home. ~ 5-6 hours

Cheers
Claudia

Scott Sumner

@Claudia-Frank

You’ve inspired me to try out some things which ended up a real “hack”. Basically I need to stop the endless chain of calls, so I measured the time gap between them (typically 4-5 milliseconds). Then I added in some code such that if an UPDATEUI is triggered within 10ms of the previous one, the new one does an immediate return, thus preventing the infinite re-triggering. It seems to work, but if you come up with something more elegant, I’d like to see it. :)

Claudia Frank

@Scott-Sumner

Hi Scott,

after analyzing the issue I think the solution is quite simple.
Note, I’m still using linux and wine - I hope it acts the same
as with real windows.

From what I see, if you have an cloned document you get additional
updateui calls, but all with update flag 0x1 and I assume those can be
safely ignored. See list of available messages and meaning.

SC_UPDATE_CONTENT       0x01 	Contents, styling or markers have been changed.
SC_UPDATE_SELECTION 	0x02 	Selection has been changed.
SC_UPDATE_V_SCROLL      0x04 	Scrolled vertically.
SC_UPDATE_H_SCROLL      0x08 	Scrolled horizontally.

So the solution I tried is

def updateui_callback(args):
    if args['updated'] == 2:
        ... do your stuff ...

seems to work. What do you think? Is it elegant ;-)

Cheers
Claudia

dail

@Claudia-Frank

That was the first thing I tried, however there are times when you need to use SC_UPDATE_CONTENT. For example putting the cursor in the middle of a word and pressing space will not re-adjust the highlighting of the word.

Claudia Frank

@dail

Hi dail, I see what you mean but I thought for Scotts purpose it might be a possible solution.
Digging deeper it seems to be a bug - but unsure what caused it (npp, scintilla).
When setting an identifier we could see, that notification are sent from the editor which has the
cursor (formerly I thought it is some kind of circular reference).
Another workaround, which keeps the SC_UPDATE_CONTENT functionality is to record
the last modification type and compare it against the updated value. But this, of course, would
mean that we can trust the ordering of the incoming notifications. From my limited tests I’ve
done, it seems that it could work.

So, for python, I did

last_mod_type = -1

def callback_sci_MODIFIED(args):
    global last_mod_type
    last_mod_type = args['modificationType']

and within updateui callback a check
to see if these nonsense notifications are receeived. If so, skip it.

def callback_sci_UPDATEUI(args):
    if args['updated'] == 1 and last_mod_type == 16400:
        return

What do you think?

Cheers
Claudia

Scott Sumner

@Claudia-Frank , @dail ,

I’ve been continuing to work this as I have time, just for “fun” but I admit it is turning away from the fun side as I discover more. Claudia’s latest fix suggestion seems to work; so it’s not that. I’ve noticed some other things which cast doubt upon the whole base concept; that being to only highlight what the user can see on screen in an editor tab.

The editor.getFirstVisibleLine() and/or editor.linesOnScreen() don’t seem to tell the whole story as to what is visible. The following at a minimum seem to mess with the accuracy of the return values of those functions: “Wrap” enabled, “Folding” in the folded-state, “Hide Lines” with lines hidden. The new “scroll beyond EOF” feature in 7.x, when enabled, causes a minor problem, but can be compensated for. But I’ve found no way to get correct screen line/position ranges with the wrap/folding/hidden features “on”. Maybe I’m missing something?

dail

The editor.getFirstVisibleLine() and/or editor.linesOnScreen() don’t seem to tell the whole story as to what is visible.

You are absolutely correct. Which is what I was alluding to in a much earlier post I stated “This isn’t perfect but works under alot of circumstances.”

Notepad++'s smarthighlighter handles this (not sure if it handles all cases you pointed out) with this section of code. It relies on the SCI_DOCLINEFROMVISIBLE message.

Scott Sumner

Forgetting all the nuances of the recent part of this thread; I’ve noticed a difference between Pythonscript and Luascript for the core part of the highlighting. Unfortunately for me, because I want to use Pythonscript, it is the Luascript version that seems to work correctly.

Consider the following Luascript code, that will highlight all of the “if” keywords that it finds within itself:

local indicator = 12 -- not sure what one is best to use but this works
editor.IndicStyle[indicator] = INDIC_ROUNDBOX
editor.IndicAlpha[indicator] = 55
editor.IndicOutlineAlpha[indicator] = 255

editor.IndicatorCurrent = indicator
editor:IndicatorClearRange(0, editor.TextLength)

local endPos = editor.TextLength
local s, e = editor:findtext('if', SCFIND_WHOLEWORD | SCFIND_MATCHCASE, 0, endPos)
while s ~= nil do
    editor:IndicatorFillRange(s, e - s)
    s, e = editor:findtext('if', SCFIND_WHOLEWORD | SCFIND_MATCHCASE, e, endPos)
end

-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if

Now consider the “equivalent” Pythonscript code:

indicator = 12  # not sure what one is best to use but this works
editor.indicSetStyle(indicator, INDICATORSTYLE.ROUNDBOX)
editor.indicSetAlpha(indicator, 55)
editor.indicSetOutlineAlpha(indicator, 255)

editor.setIndicatorCurrent(indicator)
editor.indicatorClearRange(0, editor.getTextLength())

if 0:
    def match_found(m):
        editor.setIndicatorCurrent(indicator)
        editor.indicatorFillRange(m.span(0)[0], m.span(0)[1] - m.span(0)[0])
    editor.research('if', match_found, 0, 0, editor.getTextLength())
else:
    endPos = editor.getTextLength()
    temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, 0, endPos, 'if')
    while temp != None:
        (s, e) = temp
        editor.setIndicatorCurrent(indicator)
        editor.indicatorFillRange(s, e - s)
        temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, e, endPos, 'if')

# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if

The “if 0” part of the Pythonscript code is in there so I can try it a couple of different ways; currently the “else” part of that “if 0” is active, which represents the Python code more similar to the Lua.

So…if I run the Luascript, BAM!, all of the “if” text in the source code get highlighted–no problem. However, the Pythonscript version runs a lot slower (so slow you can watch it working), and in the end not all of the “if” keyword text in its source are highlighted. During the run, some of it is highlighted temporarily, seemingly in a different color than the desired grey, even.

I’m at a loss to explain A) the difference in speed, B) why are not all of the "if"s highlighted in the end, and C) as it runs, what is going on with the different color highlighting and clearing of highlighting.

Also, changing the Pythonscript’s “if 0” to “if 1” results in similar behavior.

Any ideas on why the Pythonscript performs as it does?

Here’s my attempt to link to two animated GIFs that show both the Lua and the Python versions running:
http://imgur.com/a/2cFfG . I tried reading the “help” on the “COMPOSE” button for embedding images directly, but I didn’t understand the syntax and when I made my best guess it didn’t “preview” so I guess I had it wrong.

dail

…embedding images directly…

You can embed images like this ![some text if you want](http://i.imgur.com/eyGs0WK.gif).

Any ideas on why the Pythonscript performs as it does?

If I had to guess I’d say it comes down to the asynchronous vs synchronous execution of the scripts. I’m not familiar enough with the internals of the PythonScript to know what all gets ran asynchronously (maybe everything by default?).

Scott Sumner

@dail

Ah, the sync/async thing again. I’m sure I was thinking that if a pythonscript is run manually and not part of a callback, that it should execute correctly (since there is no way that I know of to specify sync/async when running manually).

So I wrapped the recent code in an async callback for double-click, and sure enough, it runs quickly and correctly that way (after enabling the “research” branch of the ‘if’, not the “findtext” branch–as we discovered before findtext can’t be used asynchronously). So that explains that. What it seems to indicate, however, is that a pythonscript that is run manually (from menu or shortcut keycombo) can’t be counted on to do indicators correctly, since it apparently will be run in the wrong mode.

Scott Sumner

@Scott-Sumner said:

So I wrapped the recent code in an async callback for double-click

I said this wrong. It should have been “So I wrapped the recent code in a sync callback for double-click”

and just to be totally clear the setup of the callback looks like this:
editor.callbackSync(callback_sci_DOUBLECLICK, [SCINTILLANOTIFICATION.DOUBLECLICK])

Claudia Frank

@Scott-Sumner

I think you found a bug.
Regardless of executing via callback or “manually”, the result should always be correct and this isn’t.

In regards of speed comparision, I have the feeling that lua is using some gui freeze technique
(@dail should know more about it ;-)) whilst python script seems to update every hit instantly.
In addition, the python code sets the indicator every time a new “if” is found whilst lua is doing it once. I guess this is related to the asynchronous stuff because I’ve reported this some time ago and Dave mentioned that it is most likely that the python lexer is jumping in and interrupting the
script coloring. To be honest, it is ok for me as I don’t have huge documents which need to be colored manually and a simple trick would be “unfocusing” the doc to speed things up.

Cheers
Claudia

dail

I have the feeling that lua is using some gui freeze technique

Yes and no. Since the Lua code runs synchronously there is no way that Scintilla or Notepad++ can redraw the GUI until Lua returns. So since Lua sets the indicators on multiple ranges before it returns, Scintilla only redraws once.

whilst python script seems to update every hit instantly.

If we are assuming it is actually running asynchronously (again not completely sure about the internals), then there is a fight between Scintilla and Python. Scintilla wants to redraw as soon as there is a change, but Python is running in a separate thread constantly updating the indicators. However this still doesn’t explain everything…for example why are indicators getting removed in the example gif Scott provided.

Claudia Frank

@dail

I’m not sure about this, but, can we be sure that, if a synchronous action gets called it is more or
less an atomic action? As far as I understood only the processor can assure atomic executions.
In addition I thought that plugins running in their own threads, don’t they?

for example why are indicators getting removed in the example gif Scott provided.

:-) because of the bug?? :-) I was able to confirm that there must be a bug but my results weren’t
that bad as Scotts - I assume it is related to the script runner Scott is using or …
Scott you are using python script 1.0.8.0, do you?

Cheers
Claudia