How to bookmark only the first occurrences of multiple search results?

Viktoria Ontapado

Thank you, Claudia and have a good time!

Claudia Frank

I don’t see the reported issue about the first line being empty or not,
therefore I have add two debug statements that will print the information
to the python script console. Hopefully we do find out what’s going on.

Here the new script with the fix (whole word only)
Use it as template for the three other versions and replace the for loop
as shown below only.

Script 1 (complete code)
find first occurrence of each keyword even if same line has been already found

# needed to be able to read the file as utf-8 encoded content
import codecs

def read_keyword_file(_file):
    # create the keyword file name path
    keyword_file_to_read = _file.replace('_sentences','_keywords')
    # reset _keywords variable to prevent search with wrong keywords
    _kewords = ''
    # open the keyword file as utf-8 encoded file
    with codecs.open(keyword_file_to_read, 'r', 'utf-8') as f:
        # and read line by line and create the keyword list
        _keywords = [line.strip() for line in f]
        
    # return the new created keyword list
    return _keywords


# get the complete file path of current document
_file = notepad.getCurrentFilename()

# calculate end position (=text length)
end_position = editor.getTextLength()

# read and create the keyword word list from the proper keyword file
keywords = read_keyword_file(_file)

# variable to store the content for the new file
new_file_content = ''

#replace code starting from here
# loop over the keywords
for word in keywords:
    console.write('word:{}\n'.format(word.encode('utf-8')))
    # and find each first position
    position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)
    # if we found the position of the keyword
    if position is not None:
        console.write('{}\n'.format(editor.getLine(editor.lineFromPosition(position[0]))))
        # append it to the new file content
        new_file_content += editor.getLine(editor.lineFromPosition(position[0]))
#to here

# open a new document
notepad.new()
# and add the new content
editor.addText(new_file_content)
# save it in the same directory as the original sentences file but with keytences in it.
notepad.saveAs(_file.replace('_sentences','_keytences'))


console.write('{}\nScript finished!!\n'.format(_file))

As you see it contains two comments
#replace code starting from here
and
#to here

because, instead of posting the whole script 4 times, I post the changes of the three
other scripts only.

Script 2 (only code in loop for word in keywords changed)
find first occurrence of each keyword but do not count same line more than once

# loop over the keywords
for word in keywords:
    # and find each first position
    position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)
    # if we found the position of the keyword
    if position is not None:
        # get new line
        _new_line = editor.getLine(editor.lineFromPosition(position[0]))
        # check if the line hasn't been added yet
        if _new_line not in new_file_content:
            # append it to the new file content
            new_file_content += _new_line

Script 3 (only code in loop for word in keywords changed)
find every occurrence of each keyword even if same line has been already found

#loop over the keywords
for word in keywords:
    # and find each first position
    position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)
    # if we found the position of the keyword
    while position is not None:
        # append it to the new file content
        new_file_content += editor.getLine(editor.lineFromPosition(position[0]))
        # find next position
        position = editor.findText(FINDOPTION.WHOLEWORD, position[1]+1, end_position, word)

Script 4 (only code in loop for word in keywords changed)
find every occurrence of each keyword but do not count same line more than once

#loop over the keywords
for word in keywords:
    # and find each first position
    position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)
    # if we found the position of the keyword
    while position is not None:
        # get new line
        _new_line = editor.getLine(editor.lineFromPosition(position[0]))
        # check if the line hasn't been added yet
        if _new_line not in new_file_content:
            # append it to the new file content
            new_file_content += _new_line
        position = editor.findText(FINDOPTION.WHOLEWORD, position[1]+1, end_position, word)

Python, being the scripting language, is fussy about the indention.
So make sure that you keep the format when copy and replacing the code.

Cheers
Claudia

Viktoria Ontapado

@Claudia-Frank

Thank you for your efforts till now, I go through your provided scripts before getting some sleep.

Two observations:

1.
The already mentioned first line matter is still present.
So regarding our sample, if the keyword-txt doesn’t contain an empty line, I don’t get the results with the word ‘stecken’. This issue applies to all 4 scripts.
Thanks to your debug statements I noticed a difference between the provided info on the console, so we’ve got some solid leads on this problem, I hope.

Without empty line:

word:stecken
word:besuchen
Come and see me. Komm mich besuchen.

word:die Antwort
The answer is 42. Die Antwort lautet zweiundvierzig.

word:fertig
That’s me. Ich bin jetzt fertig.

word:zuletzt
He had the last laugh. Er lachte zuletzt.

word:die Polizei
Call the police! Rufen Sie die Polizei!

word:das Glück
Luck is against me. Das Glück hat sich gegen mich gewendet.

word:auch
I’m 17, too. Ich bin auch siebzehn.

C:\Users\user\Documents\mati\German_sentences.txt
Script finished!!

With empty line:

word:
word:stecken
You’re in trouble. Sie stecken in Schwierigkeiten.

word:besuchen
Come and see me. Komm mich besuchen.

word:die Antwort
The answer is 42. Die Antwort lautet zweiundvierzig.

word:fertig
That’s me. Ich bin jetzt fertig.

word:zuletzt
He had the last laugh. Er lachte zuletzt.

word:die Polizei
Call the police! Rufen Sie die Polizei!

word:das Glück
Luck is against me. Das Glück hat sich gegen mich gewendet.

word:auch
I’m 17, too. Ich bin auch siebzehn.

C:\Users\user\Documents\mati\German_sentences.txt
Script finished!!

2.
The second script has a specific problem regarding our intent.
At the moment without empty first line I get 5, with empty first line I get 6 keytences with our sample 8-keyword base.

Working with your example sentence (Das Glück war das die Polizei die Antwort kannte.)
that means when the script arrives at the keyword “die Polizei”, it simply omits the result, just like later with “das Glück”. (Because their first occurrences are already displayed as a match for ‘die Antwort’)

I try to expand on my idea a bit again in case there is any misunderstanding.

In the case of this 2nd script, we need the same number of keytences as in the 1st script. So the 2 missing keytences shouldn’t be omitted instead as I wrote in my previous reply:

The second script behaviour should be:
[The first occurrence of die Polizie is already displayed in a result for ‘die Antwort’ keyword so it displays the second occurrence of ‘die Polizie’]
[The first occurrence of das Glück is already displayed in a result for ‘die Antwort’ keyword so it displays the second occurrence of ‘das Glück’]

Or going further with this thought, if the sentence which has the second occurrence of ‘die Polizie’ [which is the first “pure next occurence” in the sense that it only contains the word ‘die Polizei’ without containing the word ‘die Antwort’] contains the word ‘das Glück’ as well (Example: Das Glück hat ihn verlassen, die Polizei verfolgt ihn.) then in the case of the subsequent keyword “das Glück”, the following rule should be applied:

[The first occurrence of das Glück is already displayed - in a result for ‘die Antwort’ keyword; the second occurrence of ‘das Glück’ is already displayed- in a result for ‘die Polizei’ keyword; so now it displays the third occurrence of ‘das Glück’]

I hope I managed to express my idea a bit more clearly and that i don’t messed up the detailed explanation.

Apart from these two points, I didn’t notice any other issue during testing.

Thank you,
Viktória

Claudia Frank

@Viktoria-Ontapado

Hello Viktória,

Concerning the issue with the empty line I do see that this causes a problem
but my observation is a little bit different as yours, because it looks like
an empty line in keywords file causes to find the first line as match always.
Meaning, the code line

position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)

returns the position (0,0). ( I thought it returns None as if it doesn’t find anything)

Could you please change the first console.write statement with the version below
and redo the test. I expect the result is the same as before but I hope to see
a difference in the console output.

console.write('word:{}\nlength:{}\n'.format(word.encode('utf-8'),len(word.encode('utf-8'))))

In order to get rid of the empty line issue I see, replace the
keyword list creation code with this one

_keywords = [line.strip() for line in f if len(line.strip()) > 0]

Which basically means that an empty line doesn’t get added to the keyword list.

Regarding script version 2 I’m confused as it sounds you expect the output
to be the same as in script version 1. In that case there is no need for another version
if it should result in the same output.

If this isn’t your intention, could you do me a favor and create a sample
sentences file together with a keywords file and the 4 expected keytences files
and post it here?

Cheers
Claudia

Viktoria Ontapado

@Claudia-Frank

Hello,

I modified your noted lines, here are the results:

Without empty line:

word:stecken
length:10
word:besuchen
length:8
Komm mich besuchen.

word:die Antwort
length:11
Die Antwort lautet zweiundvierzig.

word:fertig
length:6
Ich bin jetzt fertig.

word:zuletzt
length:7
Er lachte zuletzt.

word:die Polizei
length:11
Rufen Sie die Polizei!

word:das Glück
length:10
Das Glück hat sich gegen mich gewendet.

word:auch
length:4
Ich bin auch siebzehn.

C:\Users\user\Documents\mati\German_sentences.txt
Script finished!!

With empty line:

word:
length:3
word:stecken
length:7
Sie stecken in Schwierigkeiten.

word:besuchen
length:8
Komm mich besuchen.

word:die Antwort
length:11
Die Antwort lautet zweiundvierzig.

word:fertig
length:6
Ich bin jetzt fertig.

word:zuletzt
length:7
Er lachte zuletzt.

word:die Polizei
length:11
Rufen Sie die Polizei!

word:das Glück
length:10
Das Glück hat sich gegen mich gewendet.

word:auch
length:4
Ich bin auch siebzehn.

C:\Users\user\Documents\mati\German_sentences.txt
Script finished!!

Script1, script3 and script4 as far as I can see working superbly - excluding our empty line issue.
Regarding the 2nd script, it’s working as well but not exactly in a way I’d like to, so at this time I try to illustrate my idea with a concrete example as you wished. I don’t know whether you can modify it accordingly, it’d be awesome though.

As usual, I’m using our sample keyword-list with 8 item:
(To ease my explanation, I assign a number one after another to every keyword):

stecken (1)
besuchen (2)
die Antwort (3)
fertig (4)
zuletzt (5)
die Polizei (6)
das Glück (7)
auch (8)

I made a sample sentencelist with 20 short examples. So this is our German_sentences.txt now:

Sie stecken in Schwierigkeiten.
Komm mich besuchen.
Stecken Sie Ihre Waffe ins Halfter!
Das Glück war das die Polizei die Antwort kannte.
Die Antwort gefällt mir.
Ich bin jetzt fertig.
Er lachte zuletzt.
Das Glück hat ihn verlassen, die Polizei verfolgt ihn.
Wann hast du sie zuletzt gesehen?
Ich liebe dich.
Ich bin auch siebzehn.
Ich bin auch achtzehn.
Rufen Sie die Polizei!
Ich bin auch zwanzig.
Wann kann ich dich besuchen?
Das Glück war ihm hold.
Mach das fertig.
Das Glück ist nicht so launenhaft.
Steht mir dieses Kleid?
Ich esse.

Your 2nd script will result the following keytences-file:

Sie stecken in Schwierigkeiten. (1)
Komm mich besuchen. (2)
Das Glück war das die Polizei die Antwort kannte. (3)
Ich bin jetzt fertig. (4)
Er lachte zuletzt. (5)
Ich bin auch siebzehn. (8)

So the script found the 1st sentence with stecken; the 1st sentence with besuchen; the first sentence with die Antwort; the 1st sentence with fertig and the 1st sentence with zuletzt. Then it omitted the result with die Polizei because its first occurence was already displayed (3); then it omitted the result with das Glück because its first occurence was already displayed (3); then found the 1st result with auch.

If it’s somehow achievable, what I’d like to get based on our sample sentencelist is this:

Sie stecken in Schwierigkeiten. (1)
Komm mich besuchen. (2)
Das Glück war das die Polizei die Antwort kannte. (3)
Ich bin jetzt fertig. (4)
Er lachte zuletzt. (5)
Das Glück hat ihn verlassen, die Polizei verfolgt ihn. (6)
Das Glück war ihm hold. (7)
Ich bin auch siebzehn. (8)

So the script find the 1st sentence with stecken; the 1st sentence with besuchen; the first sentence with die Antwort; the 1st sentence with fertig and the 1st sentence with zuletzt.

Although it omits the first result with die Polizei because that first occurence was already displayed (3); but instead of returning 0 results, it jumps to the next occurrence so we now have a sentence with die Polizei as well.
Then the script omits the first result with das Glück (because its first occurence was already displayed (3).
The next occurence in our example would be (6) but it should skip it as well because that sentence result belongs now to the keyword die Polizei and a given sentence/line shouldn’t belong to more than one keyword.
So it jumps to the next occurrence, in this case, to the third instance of das Glück so we can have its sentence (7).

(You noted that “I’m confused as it sounds you expect the output
to be the same as in script version 1.”
As you see, it’s not the case because for these three keywords [die Antwort, die Polizei, das Glück]

Script1 will give these results:

Das Glück war das die Polizei die Antwort kannte.
Das Glück war das die Polizei die Antwort kannte.
Das Glück war das die Polizei die Antwort kannte.

Script2 now will give these results:

Das Glück war das die Polizei die Antwort kannte.
NO RESULT
NO RESULT

I wish Script2 would give these results:

Das Glück war das die Polizei die Antwort kannte.
Das Glück hat ihn verlassen, die Polizei verfolgt ihn.
Das Glück war ihm hold.

I’m quite sure that I’m the reason of the misunderstading and there was a misleading, ambiguous part in my earlier posts in this regard, sorry for that.

I described my 2nd script earlier as:
“Results with only first occurences + in the case of “Das Glück war das die Polizei die Antwort kannte.” your example should be displayed in the keytence only 1 time.”
So what I meant by this is that to every keyword should belong the first, but previously not displayed occurrence somehow. So every keyword needs their own keytence which doesn’t contain a former keyword.

Probably the problem is that the script is based on the template (?i)\bKeyword\b(?s).* so for start it only selects the first bookmarked occurrences so it doesn’t have a chance to jump to a next instance?

Did I managed to explain it properly?

Claudia Frank

@Viktoria-Ontapado

quick reply in regards to empty line.
As you see, it seems there are additional chars in front of word stecken.
The correct length is 7 but it is reported as 10.
So I assume your keyword file is not utf8 but maybe utf8 with BOM !???
What does npp report in the status line (bottom line)?
The second field on the right side.
The right most (most right ??) field should be either INS or OVR.
I’m interested in the field left to INS/OVR.

Thx for the samples - I will have a look and come back on this.

Cheers
Claudia

Claudia Frank

Hello Viktória,

First let me make clear that this script is not using regular expression at all.
It just takes the keywords as strings and tries to find it in the text.

Concerning the 2nd script, I guess, now I got it.:
It should find each keyword, in a loop, until a sentences is returned which
has not been returned yet.

Which is this.

# loop over the keywords
for word in keywords:
    # and find each first position
    position = editor.findText(FINDOPTION.WHOLEWORD, 0, end_position, word)
    # if we found the position of the keyword
    while position is not None:
        # check if the line hasn't been added yet
        _new_line = editor.getLine(editor.lineFromPosition(position[0]))
        # get new line
        if _new_line not in new_file_content:
            # append it to the new file content
            new_file_content += _new_line
            break
        position = editor.findText(FINDOPTION.WHOLEWORD, position[1]+1, end_position, word)

Sometimes I do not see the wood because of the trees. :-)

Cheers
Claudia

Viktoria Ontapado

@Claudia-Frank

Bingo!
My default encoding is UTF-8 without BOM so I’m not sure what was the reason for this alteration but your guess was right, the keywords-file was in UTF-8-BOM.

I converted it to UTF-8 without BOM and now it solved the issue, emptly line no longer needed for the process to working flawlessly, thank you!

My explanation worked well as well because your modifed 2nd script does exactly what I was looking for, I’m obliged. (I’m happy anyway for that initial misunderstanding because this way I can have 5 scripts with different tasks.:-)

Finally, to clear up something:

We changed some lines due to this UTF-stuff like:
console.write('word:{}\nlength:{}\n'.format(word.encode('utf-8'),len(word.encode('utf-8')))) and
_keywords = [line.strip() for line in f if len(line.strip()) > 0]

in the scriptbase.

Now that we figured out the BOM-issue, can I return to the initial script and its variants or for safety’s sake should I rather keep the version with these modified lines? What do you suggest?

Claudia Frank

@Viktoria-Ontapado

use the current (modified) version because

the _keywords = [line.strip() …

is needed, don’t change it.

The two lines starting with console.write can be deleted if you wish but I would keep it
and comment it, (right click on the line and use context menu to comment) because
if one day something doesn’t work you just can uncomment it again and you do have
some simple debugging, which could help to find out what the cause is.

Glad to see, that it finally works and I hope you can benefit from it.

Cheers
Claudia

Viktoria Ontapado

@Claudia-Frank

All right, I follow your suggestions.

Again, I’d like to say a big thanks to you from the bottom of my heart, you provided an invaluable help for me, I’m beyond grateful.

Have a nice week,
Viktória

guy038

Hi, @viktoria-ontapado,

Sorry, but, the last two days, I was far away from my beloved laptop :-D ( Actually, it’s quite an antiquated machine !! )

I’m pleased, Viktoria, that @claudia-frank succeeded to create your four customized Python scripts. She did great work, indeed :-))

From your problem, it’s quite easy to understand that dealing with scripts is much more powerful than running a couple of regexes !

However, just for fun, I tried to imagine how to solve your case #2, with regexes ! So, in a new tab, copy the 8 regexes, corresponding to the eight keywords, followed with your 20 sentences example :

(?i-s)(?!.+#)(\bstecken\b)(?s)(.*)
(?i-s)(?!.+#)(\bbesuchen\b)(?s)(.*)
(?i-s)(?!.+#)(\bdie Antwort\b)(?s)(.*)
(?i-s)(?!.+#)(\bfertig\b)(?s)(.*)
(?i-s)(?!.+#)(\bzuletzt\b)(?s)(.*)
(?i-s)(?!.+#)(\bdie Polizei\b)(?s)(.*)
(?i-s)(?!.+#)(\bdas Glück\b)(?s)(.*)
(?i-s)(?!.+#)(\bauch\b)(?s)(.*)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sie stecken in Schwierigkeiten.
Komm mich besuchen.
Stecken Sie Ihre Waffe ins Halfter!
Das Glück war das die Polizei die Antwort kannte.
Die Antwort gefällt mir.
Ich bin jetzt fertig.
Er lachte zuletzt.
Das Glück hat ihn verlassen, die Polizei verfolgt ihn.
Wann hast du sie zuletzt gesehen?
Ich liebe dich.
Ich bin auch siebzehn.
Ich bin auch achtzehn.
Rufen Sie die Polizei!
Ich bin auch zwanzig.
Wann kann ich dich besuchen?
Das Glück war ihm hold.
Mach das fertig.
Das Glück ist nicht so launenhaft.
Steht mir dieses Kleid?
Ich esse.

Select the first regex ( (?i-s)(?!.+#)(\bstecken\b)(?s)(.*) )
Open the Replace dialog ( Ctrl+ H )
Inside the replacement zone, type in the regex \1#\2
Click on the Replace All button

=> Only one replacement is performed : A # symbol is added, right after the word stecken

Now, select the second regex (?i-s)(?!.+#)(\bbesuchen\b)(?s)(.*)
To UPDATE the Replace dialog, which is unfocused, just use, again the Ctrl + H shortcut ! ( Nice trick, indeed ! )
Click, again, on the Replace All button

=> This time, a # symbol is added at the end of the word besuchen, in the second sentence

Go on, selecting the third regex (?i-s)(?!.+#)(\bdie Antwort\b)(?s)(.*)

And so on …

Once the 8 regexes executed, you should get that text, below :

Sie stecken# in Schwierigkeiten.
Komm mich besuchen#.
Stecken Sie Ihre Waffe ins Halfter!
Das Glück war das die Polizei die Antwort# kannte.
Die Antwort gefällt mir.
Ich bin jetzt fertig#.
Er lachte zuletzt#.
Das Glück hat ihn verlassen, die Polizei# verfolgt ihn.
Wann hast du sie zuletzt gesehen?
Ich liebe dich.
Ich bin auch# siebzehn.
Ich bin auch achtzehn.
Rufen Sie die Polizei!
Ich bin auch zwanzig.
Wann kann ich dich besuchen?
Das Glück# war ihm hold.
Mach das fertig.
Das Glück ist nicht so launenhaft.
Steht mir dieses Kleid?
Ich esse.

=> The keyword matched, on each line, is easily visible, thanks to the # symbol, added to that keyword !

Finally, to delete all lines, which does NOT contain a # symbol, as well as the symbol, itself, perform the S/R :

SEARCH ^[^#\r\n]+\R|#

REPLACE Empty

Sie stecken in Schwierigkeiten.
Komm mich besuchen.
Das Glück war das die Polizei die Antwort kannte.
Ich bin jetzt fertig.
Er lachte zuletzt.
Das Glück hat ihn verlassen, die Polizei verfolgt ihn.
Ich bin auch siebzehn.
Das Glück war ihm hold.

Et voilà !

Notes :

The general regex is (?i-s)(?!.+#)(\bKeyWord\b)(?s)(.*)
As usual, the modifiers (?i-s) forces a search in an insensitive way and tell the regex engine that the dot matches a single standard character
The part (?!.+#) is a negative look-ahead, which means that an overall match implies that a # symbol cannot be found, further, on the current line
If so, the regex (\bKeyWord\b) looks for the exact word “KeyWord”, stored as group 1, due to the parentheses
Then, the modifier (?s) implies that, from now on, the dot matches any single character, even End of line characters
Finally, the part (.*) stores, as group 2, all the text, after the current keyword, till the very end of the file
In replacement, the current keyword \1 is rewritten, followed by a # character, followed, itself, by the remaining of text \2

To end, @viktoria, I hope that you’re quite aware that the order of search of the different keywords may change the sentences found by the scripts !

Indeed, if, for instance, the string das Glück is searched, BEFORE the string die Polizei, the sentence Rufen Sie die Polizei! will be found and NOT the sentence Das Glück war ihm hold. !! Yeah, not easy to get all pieces of information, in one go :-((

Best Regards,

guy038

Viktoria Ontapado

@guy038

Very impressive, guy038, thank you very much. Though I have my beautiful scripts now thanks to Claudia, I worked through your regex-based solution. Along with your notes, so much can be learnt, really.

Take care,
Viktória