[vi simulator] how to highlight a word

[Scott Sumner]

@Claudia-Frank

You’ve inspired me to try out some things which ended up a real “hack”. Basically I need to stop the endless chain of calls, so I measured the time gap between them (typically 4-5 milliseconds). Then I added in some code such that if an UPDATEUI is triggered within 10ms of the previous one, the new one does an immediate return, thus preventing the infinite re-triggering. It seems to work, but if you come up with something more elegant, I’d like to see it. :)

[Claudia Frank]

@Scott-Sumner

Hi Scott,

after analyzing the issue I think the solution is quite simple.
Note, I’m still using linux and wine - I hope it acts the same
as with real windows.

From what I see, if you have an cloned document you get additional
updateui calls, but all with update flag 0x1 and I assume those can be
safely ignored. See list of available messages and meaning.

SC_UPDATE_CONTENT       0x01 	Contents, styling or markers have been changed.
SC_UPDATE_SELECTION 	0x02 	Selection has been changed.
SC_UPDATE_V_SCROLL      0x04 	Scrolled vertically.
SC_UPDATE_H_SCROLL      0x08 	Scrolled horizontally.

So the solution I tried is

def updateui_callback(args):
    if args['updated'] == 2:
        ... do your stuff ...

seems to work. What do you think? Is it elegant ;-)

Cheers
Claudia

[dail]

@Claudia-Frank

That was the first thing I tried, however there are times when you need to use SC_UPDATE_CONTENT. For example putting the cursor in the middle of a word and pressing space will not re-adjust the highlighting of the word.

[Claudia Frank]

@dail

Hi dail, I see what you mean but I thought for Scotts purpose it might be a possible solution.
Digging deeper it seems to be a bug - but unsure what caused it (npp, scintilla).
When setting an identifier we could see, that notification are sent from the editor which has the
cursor (formerly I thought it is some kind of circular reference).
Another workaround, which keeps the SC_UPDATE_CONTENT functionality is to record
the last modification type and compare it against the updated value. But this, of course, would
mean that we can trust the ordering of the incoming notifications. From my limited tests I’ve
done, it seems that it could work.

So, for python, I did

last_mod_type = -1

def callback_sci_MODIFIED(args):
    global last_mod_type
    last_mod_type = args['modificationType']

and within updateui callback a check
to see if these nonsense notifications are receeived. If so, skip it.

def callback_sci_UPDATEUI(args):
    if args['updated'] == 1 and last_mod_type == 16400:
        return

What do you think?

Cheers
Claudia

[Scott Sumner]

@Claudia-Frank , @dail ,

I’ve been continuing to work this as I have time, just for “fun” but I admit it is turning away from the fun side as I discover more. Claudia’s latest fix suggestion seems to work; so it’s not that. I’ve noticed some other things which cast doubt upon the whole base concept; that being to only highlight what the user can see on screen in an editor tab.

The editor.getFirstVisibleLine() and/or editor.linesOnScreen() don’t seem to tell the whole story as to what is visible. The following at a minimum seem to mess with the accuracy of the return values of those functions: “Wrap” enabled, “Folding” in the folded-state, “Hide Lines” with lines hidden. The new “scroll beyond EOF” feature in 7.x, when enabled, causes a minor problem, but can be compensated for. But I’ve found no way to get correct screen line/position ranges with the wrap/folding/hidden features “on”. Maybe I’m missing something?

[dail]

The editor.getFirstVisibleLine() and/or editor.linesOnScreen() don’t seem to tell the whole story as to what is visible.

You are absolutely correct. Which is what I was alluding to in a much earlier post I stated “This isn’t perfect but works under alot of circumstances.”

Notepad++'s smarthighlighter handles this (not sure if it handles all cases you pointed out) with this section of code. It relies on the SCI_DOCLINEFROMVISIBLE message.

[Scott Sumner]

Forgetting all the nuances of the recent part of this thread; I’ve noticed a difference between Pythonscript and Luascript for the core part of the highlighting. Unfortunately for me, because I want to use Pythonscript, it is the Luascript version that seems to work correctly.

Consider the following Luascript code, that will highlight all of the “if” keywords that it finds within itself:

local indicator = 12 -- not sure what one is best to use but this works
editor.IndicStyle[indicator] = INDIC_ROUNDBOX
editor.IndicAlpha[indicator] = 55
editor.IndicOutlineAlpha[indicator] = 255

editor.IndicatorCurrent = indicator
editor:IndicatorClearRange(0, editor.TextLength)

local endPos = editor.TextLength
local s, e = editor:findtext('if', SCFIND_WHOLEWORD | SCFIND_MATCHCASE, 0, endPos)
while s ~= nil do
    editor:IndicatorFillRange(s, e - s)
    s, e = editor:findtext('if', SCFIND_WHOLEWORD | SCFIND_MATCHCASE, e, endPos)
end

-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if
-- if if if if if if if if if if if if if if if if if if if if if if if if if if

Now consider the “equivalent” Pythonscript code:

indicator = 12  # not sure what one is best to use but this works
editor.indicSetStyle(indicator, INDICATORSTYLE.ROUNDBOX)
editor.indicSetAlpha(indicator, 55)
editor.indicSetOutlineAlpha(indicator, 255)

editor.setIndicatorCurrent(indicator)
editor.indicatorClearRange(0, editor.getTextLength())

if 0:
    def match_found(m):
        editor.setIndicatorCurrent(indicator)
        editor.indicatorFillRange(m.span(0)[0], m.span(0)[1] - m.span(0)[0])
    editor.research('if', match_found, 0, 0, editor.getTextLength())
else:
    endPos = editor.getTextLength()
    temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, 0, endPos, 'if')
    while temp != None:
        (s, e) = temp
        editor.setIndicatorCurrent(indicator)
        editor.indicatorFillRange(s, e - s)
        temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, e, endPos, 'if')

# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if
# if if if if if if if if if if if if if if if if if if if if if if if if if if

The “if 0” part of the Pythonscript code is in there so I can try it a couple of different ways; currently the “else” part of that “if 0” is active, which represents the Python code more similar to the Lua.

So…if I run the Luascript, BAM!, all of the “if” text in the source code get highlighted–no problem. However, the Pythonscript version runs a lot slower (so slow you can watch it working), and in the end not all of the “if” keyword text in its source are highlighted. During the run, some of it is highlighted temporarily, seemingly in a different color than the desired grey, even.

I’m at a loss to explain A) the difference in speed, B) why are not all of the "if"s highlighted in the end, and C) as it runs, what is going on with the different color highlighting and clearing of highlighting.

Also, changing the Pythonscript’s “if 0” to “if 1” results in similar behavior.

Any ideas on why the Pythonscript performs as it does?

Here’s my attempt to link to two animated GIFs that show both the Lua and the Python versions running:
http://imgur.com/a/2cFfG . I tried reading the “help” on the “COMPOSE” button for embedding images directly, but I didn’t understand the syntax and when I made my best guess it didn’t “preview” so I guess I had it wrong.

[dail]

…embedding images directly…

You can embed images like this ![some text if you want](http://i.imgur.com/eyGs0WK.gif).

Any ideas on why the Pythonscript performs as it does?

If I had to guess I’d say it comes down to the asynchronous vs synchronous execution of the scripts. I’m not familiar enough with the internals of the PythonScript to know what all gets ran asynchronously (maybe everything by default?).

[Scott Sumner]

@dail

Ah, the sync/async thing again. I’m sure I was thinking that if a pythonscript is run manually and not part of a callback, that it should execute correctly (since there is no way that I know of to specify sync/async when running manually).

So I wrapped the recent code in an async callback for double-click, and sure enough, it runs quickly and correctly that way (after enabling the “research” branch of the ‘if’, not the “findtext” branch–as we discovered before findtext can’t be used asynchronously). So that explains that. What it seems to indicate, however, is that a pythonscript that is run manually (from menu or shortcut keycombo) can’t be counted on to do indicators correctly, since it apparently will be run in the wrong mode.

[Scott Sumner]

@Scott-Sumner said:

So I wrapped the recent code in an async callback for double-click

I said this wrong. It should have been “So I wrapped the recent code in a sync callback for double-click”

and just to be totally clear the setup of the callback looks like this:
editor.callbackSync(callback_sci_DOUBLECLICK, [SCINTILLANOTIFICATION.DOUBLECLICK])

[Claudia Frank]

@Scott-Sumner

I think you found a bug.
Regardless of executing via callback or “manually”, the result should always be correct and this isn’t.

In regards of speed comparision, I have the feeling that lua is using some gui freeze technique
(@dail should know more about it ;-)) whilst python script seems to update every hit instantly.
In addition, the python code sets the indicator every time a new “if” is found whilst lua is doing it once. I guess this is related to the asynchronous stuff because I’ve reported this some time ago and Dave mentioned that it is most likely that the python lexer is jumping in and interrupting the
script coloring. To be honest, it is ok for me as I don’t have huge documents which need to be colored manually and a simple trick would be “unfocusing” the doc to speed things up.

Cheers
Claudia

[dail]

I have the feeling that lua is using some gui freeze technique

Yes and no. Since the Lua code runs synchronously there is no way that Scintilla or Notepad++ can redraw the GUI until Lua returns. So since Lua sets the indicators on multiple ranges before it returns, Scintilla only redraws once.

whilst python script seems to update every hit instantly.

If we are assuming it is actually running asynchronously (again not completely sure about the internals), then there is a fight between Scintilla and Python. Scintilla wants to redraw as soon as there is a change, but Python is running in a separate thread constantly updating the indicators. However this still doesn’t explain everything…for example why are indicators getting removed in the example gif Scott provided.

[Claudia Frank]

@dail

I’m not sure about this, but, can we be sure that, if a synchronous action gets called it is more or
less an atomic action? As far as I understood only the processor can assure atomic executions.
In addition I thought that plugins running in their own threads, don’t they?

for example why are indicators getting removed in the example gif Scott provided.

:-) because of the bug?? :-) I was able to confirm that there must be a bug but my results weren’t
that bad as Scotts - I assume it is related to the script runner Scott is using or …
Scott you are using python script 1.0.8.0, do you?

Cheers
Claudia

[dail]

if a synchronous action gets called it is more or less an atomic action?

Yes “atomic” in regards to Scintilla and Notepad++ not able to execute it’s own code.

In addition I thought that plugins running in their own threads, don’t they?

No, by default there is no threading involved. It is as simple as a function call (e.g. Notepad++ calling a plugin’s function).

because of the bug??

Yes but indicators should only get removed when editor.indicatorClearRange() is called, and assuming the Python code is only ran 1 time, then editor.indicatorClearRange() should only ever be called once…even if it is running asynchronously.

[Scott Sumner]

@Claudia-Frank

Yes, I’m using Pythonscript 1.0.8.0 with Notepad++ 7.2.2.

When you say your results aren’t as “bad” as mine, I presume you mean that more of the text that should be highlighted remains so after a run of the Pythonscript? For me, I noticed that sometimes I get more “coverage” than I got for the run where I recorded the video, but it tends to run in spurts, meaning that I can’t predict how much coverage I’ll get but that it seems to be common to the previous run. Sometimes I’ll get very good coverage (only a few pieces of text don’t get highlighted), but I have no idea what influences this situation…

My “script runner” is simply a pythonscript that executes the active .py file, so nothing special there…

[Claudia Frank]

@dail - thank you for clarification.

Yes but indicators should only get removed when editor.indicatorClearRange() is called, and assuming the Python code is only ran 1 time, then editor.indicatorClearRange() should only ever be called once…even if it is running asynchronously.

From my test

it looks like that some found items were not colored instead of the previous are cleared as shown
by Scotts gif. To be honest, I didn’t really test if the uncolored once are reported as found - I will do so.

Cheers
Claudia

[Scott Sumner]

@Claudia-Frank

At one point I had some code in there to count the "if"s so that I could make sure they were all found by the search function…and I always saw a consistent and correct count.

[Claudia Frank]

@Scott-Sumner

I can confirm, I run it 100 times and it always reported it correctly but there is another strange thing
as soon as I put a print statement (which needs sys.stdout = console redirection) into the while loop I always get all "if"s colored. I even can now put the setIndicatorCurrent out of the loop.

else:
    endPos = editor.getTextLength()
    temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, 0, endPos, 'if')
    editor.setIndicatorCurrent(indicator)
    console.clear()
    while temp != None:
        print temp
        (s, e) = temp
        editor.indicatorFillRange(s, e - s)
        temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, e, endPos, 'if')  
print 'Done'

First I thought about a timing issue but replacing the print statement with another time consuming action brings back the issue so … ???

Cheers
Claudia

[Scott Sumner]

@Claudia-Frank

Noticed that, too…sorry I didn’t mention it. :-)

[Claudia Frank]

@Scott-Sumner, @dail

I guess I found the reason. It’s basically what we already assumed. Npp is stepping in.
During debugging I noticed, that sometime indicator was reset to value 29 (in such a case it was always 29),
which, afaik, is used by smart highlighting.
Albeit following code worked by 100 consecutive runs always, there is a potential risk that it fails again.

endPos = editor.getTextLength()
temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, 0, endPos, 'if')
editor.setIndicatorCurrent(indicator)
while temp != None:
    (s, e) = temp
    editor.indicatorFillRange(s, e - s)
    if editor.getIndicatorCurrent() != indicator:
        # print 'getIndicatorCurrent:{}'.format(editor.getIndicatorCurrent())
        editor.setIndicatorCurrent(indicator)
        editor.indicatorFillRange(s, e - s)
    temp = editor.findText(FINDOPTION.WHOLEWORD | FINDOPTION.MATCHCASE, e, endPos, 'if')

Cheers
Claudia