Bookmark multiple lines.

guy038

I tested your script and it’s working nice ! However, to my mind, it seems a bit slow on consequent files :-((

To be sure, I decided to simplify your script to its maximum ! So, the script below :

for line_nbr in range(100, 3900):
    editor.markerAdd(line_nbr, 24)

Applied to a 4000 lines file, it still took some time to bookmark these 3800 lines !?

It’s quite obvious to see the difference, compared to some native bookmark commands of N++ !

Have you got any idea about this behaviour ?

Cheers,

guy038

Scott Sumner

@guy038

Have you got any idea about this behaviour ?

I don’t really have an explanation for this, aside from the fact that Notepad++ is running compiled C++ code when it bookmarks natively, and Pythonscript is interpreted and thus has a few more “layers” to go through, slowing performance. We’ve had some similar discussions before about how looking for markers can be slow…

But I ask, is it really an issue here, for typical use? I guess I don’t have a full grasp of every possible use case’s value, but to me a typical use might be creating a selection of a few dozen lines and bookmarking them, and I wouldn’t think performance problems would be noticed in this type of scenario.

I guess I haven’t added any value in my response, to the question you asked. :-(

dail

@guy038

Have you got any idea about this behaviour ?

Neither do I, however I made a similar test with LuaScript and found I was able to mark the lines in less than ~150 milliseconds. To be clear, this is not completely a fair comparison as I am currently unable to run the PythonScript code in the same test environment (i.e. hardware, other plugins installed, N++ settings, etc). So it would be nice for someone to also try this LuaScript code:

for line_nbr = 100, 3899 do
	editor:MarkerAdd(line_nbr, 24)
end

@Scott-Sumner

We’ve had some similar discussions before about how looking for markers can be slow

Searching for markers should be quite fast (e.g. using calls such as SCI_MARKERNEXT) but as I’m not aware of the previous discussion you have had regarding this issue and maybe I am completely off topic…so please disregard if that’s the case. :)

a typical use might be creating a selection of a few dozen lines and bookmarking them

One use-case I could potentially see is marking all the lines in the file, then doing things like manually unmarking lines or ranges of lines.

Claudia Frank

As far as I understood, the main difference is, that python script uses the
SendMessage mechanism whereas Lua Script uses directFunctionPointer.

SendMessage is known to be a little bit slow if it is called often.
Just an example, the same function called 10 times in a row

@time_function()
def test():
    for line_nbr in range(100, 390):
        editor.markerAdd(line_nbr, 24)

results in

test((), {}) -> None took 3.8789999485 seconds
test((), {}) -> None took 0.276999950409 seconds
test((), {}) -> None took 1.06299996376 seconds
test((), {}) -> None took 1.46799993515 seconds
test((), {}) -> None took 2.19100022316 seconds
test((), {}) -> None took 0.131000041962 seconds
test((), {}) -> None took 0.480999946594 seconds
test((), {}) -> None took 0.243000030518 seconds
test((), {}) -> None took 1.80200004578 seconds
test((), {}) -> None took 4.5720000267 seconds

In addition, python script is releasing the GIL explicitly before each SendMessage
call, but I guess this is a minor issue “performance-wise”.

Cheers
Claudia

dail

Was curious so I did a bit more testing and wrote some C code and compiled it into a plugin:

Using a direct pointer uses on average ~10ms to set 3800 bookmarks in the file.
Using SendMessage uses on average ~17ms to set 3800 bookmarks in the file.

Note: I did this under a very uncontrolled test environment so these numbers should be seen as a rough idea how these two compare.

It would be interesting to set up better tests to compare performance between “a direct function pointer” vs SendMessage vs PythonScript vs LuaScript

Claudia Frank

@dail

nearly 2 times slower is huge but this doesn’t explain my numbers.
Did you run it multiple times? If yes, then there must be something in addition
which slows down with python script.

Cheers
Claudia

dail

@Claudia-Frank

Yes I ran it numerous times and roughly averaged out the numbers I was seeing. Again this was a very rough test I did ;) I didn’t calculate a standard deviation but it was definitely much smaller than the results you were seeing with PythonScript.

Scott Sumner

@dail said that @scott-sumner said:

We’ve had some similar discussions before about how looking for markers can be slow

Aw, cmon @dail, your memory isn’t perfect? These discussions were here. I don’t have time to revisit looking at it now, but if my memory is OK, I remember that the marker searching logic that was fast didn’t exactly correspond with how the documentation says such searching should be done, but it worked.

dail

@Scott-Sumner

Aw, cmon @dail, your memory isn’t perfect?

Oh how I wish. :)

Though that discussion was regarding indicators, this is just dealing with markers.

Scott Sumner

@dail said:

Though that discussion was regarding indicators, this is just dealing with markers.

Yea, my memory recall wasn’t perfect either, sigh…

Claudia Frank

Now I’m confused, running the same script on my old single core 1.5Ghz 32bit processor
with 32bit operating system results in

test((), {}) -> None took 0.608999967575 seconds
test((), {}) -> None took 0.733999967575 seconds
test((), {}) -> None took 0.790999889374 seconds
test((), {}) -> None took 0.677000045776 seconds
test((), {}) -> None took 0.704999923706 seconds
test((), {}) -> None took 0.678999900818 seconds
test((), {}) -> None took 0.878000020981 seconds
test((), {}) -> None took 0.634999990463 seconds
test((), {}) -> None took 0.706000089645 seconds
test((), {}) -> None took 0.753000020981 seconds

Previous result was an old dual core 2.0Ghz 64bit processor on 64bit operating system, but
using 32bit notepad++ (wine api layer issue??)

Cheers
Claudia

Claudia Frank

and now I’m …

lua_markerAdd took 0.11399999999958 seconds
lua_markerAdd took 0.11599999999999 seconds
lua_markerAdd took 0.11799999999948 seconds
lua_markerAdd took 0.12899999999991 seconds
lua_markerAdd took 0.11400000000049 seconds
lua_markerAdd took 0.11599999999999 seconds
lua_markerAdd took 0.11499999999978 seconds
lua_markerAdd took 0.11400000000049 seconds
lua_markerAdd took 0.11400000000049 seconds
lua_markerAdd took 0.11499999999978 seconds

py_markerAdd((), {}) -> None took 7.81100010872 seconds
py_markerAdd((), {}) -> None took 7.80900001526 seconds
py_markerAdd((), {}) -> None took 7.68000006676 seconds
py_markerAdd((), {}) -> None took 8.02199983597 seconds
py_markerAdd((), {}) -> None took 7.70700001717 seconds
py_markerAdd((), {}) -> None took 7.73000001907 seconds
py_markerAdd((), {}) -> None took 7.84200000763 seconds
py_markerAdd((), {}) -> None took 7.71900010109 seconds
py_markerAdd((), {}) -> None took 7.76399993896 seconds
py_markerAdd((), {}) -> None took 7.83399987221 seconds

tested on same machine, with marking lines 100 to 3899

I mean, two times faster can be explained but 80 times faster.
There must be something specific which slows down the execution.

Cheers
Claudia

Claudia Frank

by using direct function pointer it is a little bit faster ;-)

DirectFunction took 0.111000061035 seconds
DirectFunction took 0.147000074387 seconds
DirectFunction took 0.148999929428 seconds
DirectFunction took 0.146000146866 seconds
DirectFunction took 0.154000043869 seconds
DirectFunction took 0.271999835968 seconds
DirectFunction took 0.24799990654 seconds
DirectFunction took 0.164999961853 seconds
DirectFunction took 0.164999961853 seconds
DirectFunction took 0.171000003815 seconds

Cheers
Claudia

Claudia Frank

and with some performance optimization, same test, same machine,

DirectFunction took 0.0729999542236 seconds
DirectFunction took 0.0710000991821 seconds
DirectFunction took 0.0720000267029 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0729999542236 seconds
DirectFunction took 0.0769999027252 seconds
DirectFunction took 0.0739998817444 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0759999752045 seconds

Enough performance testing from my side :-)
Still curious why wrapped call is so slow.

Cheers
Claudia

guy038

Hello, @dail, @scott-sumner, @claudia-frank, and All,

Ah, I didn’t think that my slowness’s feeling, regarding that script, would lead to so many posts ;-))

I realize that I was quite vague, about the results of the test. So I did, again, the test with both :

Python v1.0.8.0
Lua v0.6.0.0

installed, in a N++ v7.4.2 local configuration, on a old Win XP SP3 platform, with 1Go of RAM, only !

The python test script was :

for line_nbr in range(100, 12400):
    editor.markerAdd(line_nbr, 24)

And the Lua test script was :

for line_nbr = 100, 12400 do
    editor:MarkerAdd(line_nbr, 24)
end

Results if applied on a classical .txt file, containing 12436 lines, with size = 613,527 bytes :

1m and 59s for the Python script
0.5s, approximatively, for the Lua script

Really awesome, isn’t it ?! How to explain such a difference ??

BTW, Dail, I ran the command, below, in the Lua Console. Not sure, however, that it’s the right syntax !

dofile("Dail_Bookmark.lua")

Now, Dail, an other problem !

My first idea, for doing the test, was to install the Lua last version ( 0.7.1.0 ). Unfortunately, when starting N++, I got a tiny window, with the message Unknown exception. After validation, this windows disappeared but the lua plugin was not uploaded ( Quite sure that uploaded is not the right English word but I hope I will be understood ! ) That issue also occurs with the v0.7.0.0 version ! So, the last Lua plugin correct version is v0.6.0.0, on my XP machine.

Probably, this happens on Win WP platforms, exclusively ? But, dail, it’s not that important. Indeed, Don Ho, seemingly, considers not supporting, soon, Win XP, anymore ! Refer :

https://notepad-plus-plus.org/community/topic/15003/notepad-7-5-4-release/26

Nevertheless, thanks, by advance, for any investigation, on that matter !

Best Regards,

guy038

Update :

I’ve just done the test, twice, again, with the Python script ==> A bit better : 1m and 33s , approximatively !!

dail

@guy038

Some very interesting results for sure, I can’t say much as to why the PythonScript version is so much slower and I am very unfamiliar with the Python/C API. Maybe someone more knowledgeable has some ideas.

Unfortunately, when starting N++, I got a tiny window, with the message Unknown exception.

Sorry to hear the newer versions are not working for you, and I agree it is probably related to Windows XP. Between versions 0.6.0 and 0.7.0 I did update to VS2015 and it also uses a newer runtime. Even though it is suppose to be compatible with XP there still seems to be a problem as you have found out. Personally I do not have access to an XP machine any more so any XP related issues will probably go unfixed unless someone submits a pull request fixing them.

Claudia Frank

I don’t think file size matters, I did a test with a 100mb file as well as with
file containing just enough carriage returns -> result is the same.

I haven’t found out yet why PythonScript(PS) is so slow in this case and it must be
something how PS handles the wrapped call because when using PS and the
Direct_Function call then I do get results like this

DirectFunction took 0.0729999542236 seconds
DirectFunction took 0.0710000991821 seconds
DirectFunction took 0.0720000267029 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0729999542236 seconds
DirectFunction took 0.0769999027252 seconds
DirectFunction took 0.0739998817444 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0759999752045 seconds
DirectFunction took 0.0759999752045 seconds

Cheers
Claudia

dail

@Claudia-Frank

I did some poking around…and installed PythonScript (I haven’t used it in years). When I ran some test code, one of the first things I realized is that it is that Scintilla is being redrawn multiple times when using PythonScript. I verified this by catching the SCN_UPDATEUI event in Lua. The LuaScript call generates 1 event, using PythonScript generates numerous events…not always the same amount (normally about 50% of however many bookmarks were added). It probably also generates numerous SCN_MODIFIED events?

You might try a similar test with PythonScript which doesn’t generate any type of side effect, something simple like SCI_STYLEGETFORE which should be a very fast and simple call.

Claudia Frank

@dail

thanks for looking into this.
I will give it a try and let you know.

Cheers
Claudia

Claudia Frank

@dail

Good job, I guess that is the area which needs investigation, yes.
I tested with synchronous and asynchronous callbacks - result was more or less the same.

If I
added markers for 30 lines it report one aditional updateui event,
adding markers to 290 lines it added 3 additional event
adding markers to 3800 lines it added 40 additional events

lua reported only 1 event in all cases, the same is true if using Direct_Function call.

Cheers
Claudia