Search++ (\W)'(\w) regex replace failure

guy038

Hi, @m-andre-z-eckenrode, @alan-kilborn, @coises, @mpheath, @thomas-knoefel and All,

@coises, personally, I confirm that the step by step replacement with the regex pattern :

FIND (\W)'(\w)
REPLACE \1’\2

Works correctly with both your two plugins Columns++ and Search++ and with the MultiReplace plugin as well, when using the N++ v8.9 release !

The bug seems to occur only with native N++ regex engine !

BR

guy038

M Andre Z Eckenrode

@Coises said:

Would you try and see if you can get the search in Columns++ to misbehave under the same circumstances as the one in Search++?

In installed NPP, using Columns++ search in regex mode with a rectangular selection of multiple copies of my previously shown example text, I find that only the instances of “ 'r” (preceded by a regular space, NOT by any EOL) are matched, and are all replaced correctly with no anomalous characters. To be clear, this, again, is the starting text:
this
'Cos I am
that

this
'Round and 'round
that

this
'Cos I am
that

this
'Round and 'round
that
And resulting text:
this
'Cos I am
that

this
'Round and ’round
that

this
'Cos I am
that

this
'Round and ’round
that

M Andre Z Eckenrode

@guy038 said:

The bug seems to occur only with native N++ regex engine !

I’m confused. You’re saying that a native N++ regex replacement, using my example text and regex strings, results in anomalous characters for you? I haven’t seen that at all.

guy038

Hello, @m-andre-z-eckenrode, @coises and All,

@m-andre-z-eckenrode, you said :

I’m confused. You’re saying that a native N++ regex replacement, using my example text and regex strings, results in anomalous characters for you? I haven’t seen that at all.

I didn’t say that anormalous characters appear. I just said that no replacement occurs at all !

Indeed, with the native regex N++ replacement, and the text below :

this
'Cos I am
that

this
'Round and 'round
that

and with the following REPLACEMENT :

FIND (\W)'(\w)
REPLACE \1’\2

The first two replacements do not occur ! Only the last one will change the normal quote ' into the ’ character ( \x{2019} RIGHT SINGLE QUOTATION MARK )

IMPORTANT : This test was done with N++ v8.9. May be, the last v8.9.7 release would produce a different result !

Best Regards,

guy038

Coises

@M-Andre-Z-Eckenrode said:

using Columns++ search in regex mode with a rectangular selection of multiple copies of my previously shown example text, I find that only the instances of “ 'r” (preceded by a regular space, NOT by any EOL) are matched

I’m sorry… I didn’t think to mention that quirk. If you start with a rectangular selection, Columns++ searches each row independently, so the ones with ' at the beginning of a line would fail to match.

If Auto set is checked (the default) in the Search in indicated region dialog and no region is already indicated, when you start a search with nothing selected the indicated region is set to include the whole document. (You can also select the entire document, or make any multiple line selection or multiple selection, before you search.¹) This is different from a rectangular selection converted to an indicated region: rectangular selections are multiple selections which never include line endings.

(Yet another complication is that Scintilla does not reliably restore indicators (marked text) on undo. I think this only affects cases where a replacement begins at the first character of a contiguous span of marked text — Scintilla restores the text, but not the marker, on undo.)

So it is expected that only occurrences within a single line would match if you start with a rectangular selection. If you start with the entire document selected, or with no selection at all, it should match all the same occurrences as Search++ (with no selection or marked region).

¹ The rules are complicated. The confusing nature of the “indicated region” was one of the reasons I wanted to split Search++ entirely instead of doing more work on the search in Columns++. I didn’t want to change the concept for people who are already used to Columns++ search, but outside of the specific case of searching in column selections, I think it is more confusing than it needs to be.

Coises

@guy038 said:

I didn’t say that anormalous characters appear. I just said that no replacement occurs at all !

You might be confusing @m-andre-z-eckenrode’s topic (\W)'(\w) Regex replace failure, about Notepad++, and this topic, about Search++, with the same search.

He was surprised that Notepad++ found the text but would not replace it. The cause is how Notepad++ treats CRLF pairs combined with how it decides whether Replace should replace or find next — as I wrote there, whether it is a bug is, I suppose, a matter of opinion.

Search++ has a different potentially user-unfriendly behavior: with different find or replace expressions (any regular expression that can match starting with an LF, combined with a replacement string that doesn’t copy the first matched character as the first character of the replacement) a user could invisibly change line endings from CRLF to just CR (though if the user understands the regular expression and replacement entered, it would be expected).

Notepad++ has the same behavior as Search++ when using Replace All (including that it can replace the LF in a CRLF pair with something else); it’s just step-by-step replace that can find but fail to replace (for the same reason that \K doesn’t work in step-by-step native searches).

In this topic, the problem is that Search++ is, apparently randomly, replacing with garbage instead of the correct replacement string.

I have not yet found a possible cause. It does not happen on my machine.

M Andre Z Eckenrode

@Coises said:

I didn’t think to mention that quirk. If you start with a rectangular selection, Columns++ searches each row independently

Aha. That explains my own confusion. That was actually the first time I’d used the Search feature of Columns++. For the record, any time I’m doing regex search & replace in NPP that either turns out to be more complicated than I expected, or involves some trial and error, or which I’m going to want to utilize again elsewhere (such as this topic), I’m in the habit of typing it out right in the text document I’m working on, and then I select/copy the replacement regex, then select the find regex and open the necessary dialog, expecting the Find what to have been automatically populated from my most recent selection (which, of course, didn’t happen in that case). It didn’t occur to me that that wouldn’t work in Columns++. So, when I opened Columns++’s Search dialog with only the content of one line selected and tried to perform my regex search, it threw this at me:

This command requires a rectangular selection. Extend selection to the last line of the document?

And, Columns++ Search neophyte that I am, I didn’t heed the suggestion. I closed the dialog and created a rectangular selection.

Now that I’ve been made aware of the error of my ways, I’ve redone the experiment with all the my example text selected, and all instances of ' were successfully replaced with ’, without any anomalies.

Coises

@M-Andre-Z-Eckenrode said:
This command requires a rectangular selection. Extend selection to the last line of the document?
And, Columns++ Search neophyte that I am, I didn’t heed the suggestion.

Had you heeded it, you would still have gotten a rectangular selection, as it says, extending downward from your selection — not necessarily enclosing the whole document, and in any case not the same as selecting the whole document (or selecting nothing at all) first.

I should re-word that message, I’m just not sure what to say instead. It’s not actually true (since version 0.8) that a rectangular selection is required; as I mentioned before, the rules are complicated. They won’t easily fit into a message box. It makes sense when you’re using Columns++ for working with columns, but the search kind of took on a life of its own, apart from column work… hence, among other reasons, Search++.

I’ve redone the experiment with all the my example text selected, and all instances of ' were successfully replaced with ’, without any anomalies.

Thank you for that information. I do appreciate the help you’ve given me with this. While it doesn’t prove anything (almost nothing is ever proven when dealing with an intermittent bug), it suggests pretty strongly that something I changed in moving the Columns++ search process to Search++ has introduced a hidden instability. I’m still looking for it.

M Andre Z Eckenrode

@Coises

On the outside chance that specifics of the variants of my regex replacement anomalies are useful, here is some detailed information about them and the circumstances of their appearance… I started with an unsaved ANSI text tab/buffer with:

Search++ Regex

(\W)'(\w)
\1’\2

this
'Cos I am
that

this
'Round and 'round
that

------------------------------------------------------------------

… and the example text (this[CRLF]'Cos to 'round[CRLF]that followed by [blank line]/[dashed line]/[blank line]) repeated 9 times below (total of 10 times). The anomalies occurred on the following lines during this first experiment:

Line 27:
-’-os I am

Lines 51–52:
’Round and
’hound

Lines 62–63:
a’
ound and ’round

Second experiment:

Line 11:
-’-ound and ’round

Line 27:
-’-os I am

Line 57:
-’-os I am

Lines 87–88:
[blank line]
’tos I am

Third experiment:

Line 10:
this-’-ound and ’round

Line 16:
-’-os I am

Line 40:
-’-ound and ’round

Line 60:
’Round and-’-ound

Line 90:
’Round and-’-ound

Coises

@M-Andre-Z-Eckenrode said:

here is some detailed information

Thank you. At this point I am still stumped as to how to find this needle in a haystack. Any information could turn out to be helpful. I appreciate your perseverance.

Coises

@M-Andre-Z-Eckenrode:

Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?

M Andre Z Eckenrode

@Coises said:

Have you ever noticed arbitrary junk character replacements with any other regular expression / replacement pairs, or only with this particular one?

I haven’t actually tried any others with Search++, so far. The vast majority of my regex operations are accomplished via PythonScript, and I only very infrequently need to perform a standalone regex. But I’ll keep Search++ in mind if/when another one becomes necessary.