working use-case for SCI_REPLACETARGETRE

PeterJones

Trying to fix my “PerlScript” implementation of editor->replaceTargetRE, and trying to use a PythonScript as proof of concept.

I started with https://community.notepad-plus-plus.org/topic/11804/, and added prints and sleeps (so I can see what’s going on… not because the script needed delay), but it seems to be replacing the whole text with great instead of just replacing the beautiful

# encoding=utf-8
"""
https://notepad-plus-plus.org/community/topic/11804/ with LuaScript/PythonScript
"""
from Npp import *
from time import sleep

console.show()
console.clear()

notepad.new()
editor.setText("""This is a not selected line !!!
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
This is a not selected line !!!
""")

editor.setTargetRange(32,105)           # use hardcoded range instead of editor.targetFromSelection
console.write("-----\n"+editor.getTargetText()+"=====\n")   # added for debug
console.write("target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
if False:
    editor.replaceTarget("HAHAHA!\r\n")
    console.write("laughing target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
elif True:
    editor.searchInTarget(r'beautiful')
    console.write("beautiful target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
    sleep(1)
    editor.replaceTargetRE('great')
    console.write("great target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
    sleep(1)

console.write("final target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
console.write("-----\n"+editor.getTargetText()+"=====\n")   # added for debug

# pretend that nothing has changed, and close the file without saving
sleep(5)
editor.setSavePoint()
notepad.close()

And the console output contains:

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,137)
great target=(0,5)
final target=(0,5)
-----
great=====

The first getTargetText results between ---- and ===== are what I expect. But I would have thought that the “great target” would have been about the same size as the “beautiful target” (maybe a few chars shorter because length(great)=5 < length(beautiful)=9. Instead, it drops to a length of 5, and the whole final text is great without any newlines and without any of the original text.

So now I have what looks like a bug or misunderstanding in the PythonScript, let alone being able to debug my source code for PerlScript. :-(

Debug Info:

Notepad++ v7.8.5   (64-bit)
Build time : Mar  3 2020 - 17:39:56
Path : C:\usr\local\apps\notepad++\notepad++.exe
Admin mode : OFF
Local Conf mode : ON
OS Name : Windows 10 Enterprise (64-bit) 
OS Version : 1903
OS Build : 18362.720
Plugins : ComparePlugin.dll LuaScript.dll MarkdownViewerPlusPlus.dll mimeTools.dll NppConsole.dll NppConverter.dll NppEditorConfig.dll NppExec.dll NppExport.dll NppFTP.dll PreviewHTML.dll PythonScript.dll QuickText.dll XMLTools.dll

with PythonScript v1.5.2

Can anyone confirm that my script does the same for them?

And any suggestions for a example of a working editor.replaceTargetRE() example sequence?

Alan Kilborn

@PeterJones

This is what your script outputs for me (slightly different than yours):

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(64,69)
final target=(64,69)
-----
great=====

Ekopalypse

My result

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(64,133)
final target=(0,133)
-----
This is a not selected line !!!
This is line one !!!
Today is a great day !!!
This is line three !!!
This is a not selected line !!!
=====

but I had a different result the first time I did it. Hmm …

Alan Kilborn

I get the same results each time, but with the sleep calls in there I get a weirdness: My normally transparent Find window briefly goes non-transparent while the script runs.

Ekopalypse

Ok - I guess I got it.
Running the script as provided, well I’ve commented the sleeps, I get this

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,137)
great target=(0,5)
final target=(0,5)
-----
great=====

I didn’t expected but I thought I know where the issue is.
setSearchFlags have not been set. So I did this

...
elif True:
    editor.setSearchFlags(FINDOPTION.REGEXP)
    editor.searchInTarget(r'beautiful')
    console.write("beautiful target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
    # sleep(1)
    editor.replaceTargetRE('great')
    console.write("great target=({},{})\n".format(editor.getTargetStart(),editor.getTargetEnd()))
    # sleep(1)
...

and run the script which returned

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(0,137)
great target=(0,5)
final target=(0,5)
-----
great=====

Was confused did something different and tried it again,
same script no other modification, result

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(64,133)
final target=(0,133)
-----
This is a not selected line !!!
This is line one !!!
Today is a great day !!!
This is line three !!!
This is a not selected line !!!
=====

Not sure why this happens, but I’m more or less sure that
in order to get the target search/replace done like one wants to have
I would always set the search flags explicitly.

Btw. this has been written by me - not by deepl
I just gave it a try. :-)

Ekopalypse

But the result is wrong anyway, isn’t it?
From the documentation

SCI_REPLACETARGETRE(position length, const char *text) → position

This replaces the target using regular expressions. If length is -1, text is a zero terminated string, otherwise length is the number of characters to use. The replacement string is formed from the text string with any sequences of \1 through \9 replaced by tagged matches from the most recent regular expression search. \0 is replaced with all the matched text from the most recent search.
After replacement, the target range refers to the replacement text. The return value is the length of the replacement string.

So the expected result should be

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(64,69)
final target=(64,69)
-----
great=====

Right?

PeterJones

@Ekopalypse said in working use-case for SCI_REPLACETARGETRE:

But the result is wrong anyway, isn’t it?
So the expected result should be …
Right?

I was about to say no, that I expected to see what @Ekopalypse showed, with the full text printed out the second time… but then I remembered I’m printing the target text, which should just be the great, as you showed. I would expect the full notepad.getText() printout (not in my code, but it should have been) to be what @Ekopalypse showed.

I’ll try the explicit setSearchFlags later – hopefully in an hour or two – and report back results when I’ve had a chance. Thanks for the help.

(Trying to implement/verify calls to methods I’ve never used can be difficult, because I don’t know whether it’s me or my software that’s misbehaving. :-) )

Ekopalypse

@PeterJones

Peter you are confusing me :-D

… as you showed …

but then

… to be what @Ekopalypse showed …

I’ll try the explicit setSearchFlags later – hopefully in an hour or two – and report back results when I’ve had a chance.

Too late for me - I go to bed :-)

A good night and stay safe and healthy to everyone

PeterJones

@Ekopalypse said in working use-case for SCI_REPLACETARGETRE:

@PeterJones
Peter you are confusing me :-D

I’m confused. For some reason, I thought the most recent had been another Alan reply, rather than two Eko’s in a row. Sorry.

Too late for me - I go to bed :-)

Maybe I should to, but since the sun is still up and I haven’t had dinner yet, that’s probably not wise. ;-)

PeterJones

@Ekopalypse said

editor.setSearchFlags(FINDOPTION.REGEXP)

So when I add that line, and comment out the sleeps, it goes back and forth between your two output pairs. If I add in the

console.write(".....\n"+editor.getText()+":::::\n")

after the final getTargetText(), it duplicates whatever was printed above.

first

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(64,69)
final target=(64,69)
-----
This is a not selected line !!!
This is line one !!!
Today is a great day !!!
This is line three !!!
This is a not selected line !!!
=====
.....
This is a not selected line !!!
This is line one !!!
Today is a great day !!!
This is line three !!!
This is a not selected line !!!
:::::

second

-----
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
=====
target=(32,105)
beautiful target=(64,73)
great target=(0,5)
final target=(0,5)
-----
great=====
.....
great:::::

If I have the sleep(1) between beautiful and great, it always does the second. But the first is closer to what I would expect (except that the final getTargetText should be great, not the full text)

This is weird.

It’s going to make it really difficult to see if I’ve implemented correctly in Perl. :-)

Ekopalypse

I was using a clean 7.8.4 portable, disabled all Auto* features such as smart highlighting, auto-complete, etc. of npp and even disabled the 3 built-in plugins, and it happened anyway.

Then I used PySnack and ran it ~100 times and it never happened.
So I suppose the reason why PS has this problem is that PS uses threads when executing scripts, which is not the case with PySnack.
And because npp itself changes the target for different purposes, I assume that there is a conflict while PS is still running.

It’s going to make it really difficult to see if I’ve implemented correctly in Perl.

Tell me about it. :-)

Alan Kilborn

@Ekopalypse said in working use-case for SCI_REPLACETARGETRE:

because npp itself changes the target for different purposes,

Methinks you are onto something here.

PS uses threads when executing scripts, which is not the case with PySnack

Is now the time/place to go into how PySnack is going to be better than PythonScript? :-)
A discussion of the different architectures?

I mean, I presume “script” does things with threads because it needs to? And you decided “snack” doesn’t need to?

What’s better in “snack”? What’s “worse”?

How are things like NOTIFICATIONs and synchronous / asynchronous callbacks going to work with “snack”, without threading (maybe making it seem like I know more than I do here, hehehe, maybe one thing has nothing to do with the other).

I’m definitely in favor of the KISS principle, where it works.

Maybe a separate thread for this type of discussion, but maybe you are not even ready for that yet! :-)

BTW, unfortunate that both plugins could be abbreviated as “P.S.”. :-(

Ekopalypse

The biggest and in my opinion the only advantage of threads is that
Npp itself can’t be disturbed by PS.
Even if someone has programmed an endless loop, npp should,
most of the time, still be able to terminate itself cleanly.
The disadvantage of threads can be seen in this problem.
There is the possibility that 2 or more threads work on the same thing
and cause problems. Scintilla, to my knowledge, is not thread-safe.

PySnack does more or less the same thing as PythonScript.
Except for the obvious, i.e. using Python3 and Cython.
Boost::Regex is, unfortunately, also implemented a bit differently - but I’m not 100% sure if I keep it that way.

At first, threading has nothing to do with the notifications but on the
second view it does if we look at the asynchronous callbacks from PS.
Actually, it works like this.

Npp has a NotificationHandler that is called from Scintlla.
Any plugin can register via the beNotified Callback function,
methods which are called then.

This normally runs synchronously.

But if I now know that I need longer to calculate something,
then my plugin, kindly give this task to a separate thread and
thus return control to Npp more quickly, so nothing is blocked.
But now this thread should not make any changes to scintilla but
inform the actual main thread, that something has to be done.
So this is the theory - sometimes easier, mostly more difficult to implement.

What is worse or better with PySnack is like with art in the eye of the beholder.
I think that, at the moment, almost everything that PS has, PySnack can do too and a little bit more.
But to be honest, it is always easier to create something new from an
existing source then to invent it intially.
So without PS there would never have been the idea to develop PySnack.

BTW, unfortunate that both plugins could be abbreviated as “P.S.”. :-(

Maybe not that bad at all, - switching over could be easier :-)
One of the goals of PySnack is to be able to run the scripts of PS
with almost no changes. Won’t be 100%, but I hope 99% are in.

@PeterJones - sorry for kinda high jacking your thread.

PeterJones

@Ekopalypse said in working use-case for SCI_REPLACETARGETRE:

sorry for kinda high jacking your thread.

Not a problem.

Ran some more experiments this morning: I upgraded from PythonScript 1.5.2 to 1.5.3 (didn’t really change the behavior, but I thought it was worth mentioning). I played around with a sleep between when I generate the initial text and when I do the setSearchFlags – either before or after the setTargetRange.

If I have no sleep, then I get unrepeatable results, where sometimes the final getText is only great, and sometimes it’s the whole paragraph, with great rather than beautiful (as I expect). If I put a small sleep (whether before or after setTargetRange), the final getText is much more reliable. If I use 10ms (ie, sleep(0.01)), it was at least 10/10 reliable; if I drop to 1ms, it seemed better than 50% reliable, but it wasn’t every run; if I have no wait, getText shows just great many/most runs.

I cleaned out some of the debug stuff, and added a loop with a counter for how many runs it got right, and tried a few different delays:

# encoding=utf-8
""" https://community.notepad-plus-plus.org/topic/19217/working-use-case-for-sci_replacetargetre

I am working on implementing editor.replaceTargetRE in Win32::Mechanize::NotepadPlusPlus::Editor,
but it's currently not working.

Found https://notepad-plus-plus.org/community/topic/11804/ with LuaScript/PythonScript examples that _should_ work
"""
from Npp import *
from time import sleep

console.show()
#console.clear()

tWait = 0.005
N = 100
right = 0
srctxt = """This is a not selected line !!!
This is line one !!!
Today is a beautiful day !!!
This is line three !!!
This is a not selected line !!!
"""
cmptxt = """This is a not selected line !!!
This is line one !!!
Today is a great day !!!
This is line three !!!
This is a not selected line !!!
"""

for x in range(N):
    notepad.new()
    editor.setText(srctxt)
    sleep(tWait)

    editor.setTargetRange(32,105)

    editor.setSearchFlags(FINDOPTION.REGEXP)
    editor.searchInTarget(r'beautiful')

    editor.replaceTargetRE('great')

    if editor.getText() == cmptxt:
        right = right + 1

    editor.setSavePoint()
    notepad.close()

console.write("Final Score: tWait={} gave {}/{} right!\n".format(tWait,right,N))

which gives, on my various runs:

Final Score: tWait=0.001 gave 30/100 right!
Final Score: tWait=0.01 gave 100/100 right!
Final Score: tWait=0.002 gave 67/100 right!
Final Score: tWait=0.003 gave 95/100 right!
Final Score: tWait=0.004 gave 99/100 right!
Final Score: tWait=0.005 gave 100/100 right!
Final Score: tWait=0.005 gave 99/100 right!

So, somewhere between 5-10ms, it reliably replaces just the expected text, rather than the whole file. At least for me, with PythonScript v1.5.3 (in 64bit Notepad++ v7.8.5), on the machine I’m currently using. Smaller than that, it appears there’s a race condition, where Scintilla gets ahead of itself (tries to do the search before it’s finished writing its buffer?). Given that most of my professional programming involves programming electronics to generate and measure voltages and other signals, I am so used to adding delays in my software to avoid non-deterministic delays on the order of 1-10ms, it seems rather natural to me to add delays to my code – but I can see why someone who doesn’t come from the world of programming physical devices might balk at adding delays to code.

Also, given that the topic from 2016 had to do with oddities in the final Target location after the replacement, I am not going to worry about that. I think this sequence of setText, wait, setTarget, setSearchFlag, searchInTarget, replaceTargetRE seems reliable enough now in PythonScript that I can use it to help debug my PerlScript version.

Thanks for the hints, confirmations, and experiments.

Alan Kilborn

@PeterJones

Since, as @Ekopalypse pointed out, Notepad++ itself uses the “target” concept, and apparently(?) there can only be one of these that Scintilla knows about, isn’t any use of target-related functions (be it in Pythonscript or your Perl-based thing) going to be potentially problematic when a user tries to use it?

I know you are just trying to get thru implementing/testing/documenting of the function under discussion, but…

PeterJones

@Alan-Kilborn said in working use-case for SCI_REPLACETARGETRE:

isn’t any use of target-related functions going to be potentially problematic

Probably true. I think I will add a caveat to the “target” documentation that explains that Notepad++ might interfere with the Scintilla targets, and suggest they code defensively when using targets.

Ekopalypse

@Alan-Kilborn said in working use-case for SCI_REPLACETARGETRE:

isn’t any use of target-related functions problematic

I would say no, if it is called from the main thread or better the thread
which created the scintilla window.

PeterJones

@Ekopalypse said in working use-case for SCI_REPLACETARGETRE:

if it is called from the main thread or better the thread
which created the scintilla window.

PerlScript is an outside-looking-in application, unlike most plugin development (which starts inside). So I won’t have that, which will make it more important for me to warn users (here and elsewhere, really) that Notepad++ is still running, and could still be influencing things separately from PerlScript.

When I get to v2, with a full plugin that uses my perl module from the inside of Notepad++, I might have better control of such things.

dail

Disclaimer: I did not fully read every single line of this thread…but hopefully I can contribute a bit of helpful info.

Accessing Scintilla on a background thread can be very problematic. There are Scintilla calls that have temporal coupling and since you cannot guarantee any two calls made from PythonScript (or outside of Notepad++) are executed without being interrupted by another thread, then weird things can happen. Even doing something simple like:

editor.setTargetRange(0, 100)
x = editor.getTargetText()

could potentially give unexpected results. Notepad++ does its own work (search/replace, function list, auto completion, smart highlighting, URL highlighting, etc) which changes the target quite frequently.