Search++ (\W)'(\w) regex replace failure

[Coises]

@M-Andre-Z-Eckenrode said:

@Coises

And just now tried Find (\w)a(\w) and Replace \1z\2 with the same list of words, which definitely resulted in at least one anomalous replacement (average became avevzie), possibly more, but the mistakes are more difficult to spot at a glance in this case.

I hate continuing to bother you about this with different questions, but I am so stumped. Have you noticed:

1. Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI? (For a test, just use Encoding | Convert to UTF-8 on a document containing your test text before trying to Replace.)

2. Have you ever seen a bad replacement where the replacement string did not use a capturing group reference (like \1 or \2 or $1 or $2)?

[M Andre Z Eckenrode]

@Coises said:

Do bad substitutions happen if the document is set as UTF-8, or is it specific to ANSI?

I’ve experienced it with both at this point.

Have you ever seen a bad replacement where the replacement string did not use a capturing group reference

Not that I’ve noticed so far.

New test:

I used the same list of words, but inserted the number 2 between the second and third letters of all words that consisted of at least three letters, then stepped through using Find ([a-z])\d([a-z]) and Replace \1\2. There are anomalies. Examples include:

ac2ademic became adjdemic

ad2ministration became a
ginistration

[Coises]

@M-Andre-Z-Eckenrode

I think I may have found the cause.

Would you try installing this version:

https://github.com/Coises/SearchPlusPlus/releases/tag/v0.6.3.2

and see if the problem goes away?

If the problem is what I think it is, this will make it go away. It isn’t a good solution (it will cause serious inefficiency for large files), but if you no longer see the problem with this version, I’ll know I’ve almost certainly identified the cause, and I can work on a proper resolution.

Thank you so much for all your effort.

[M Andre Z Eckenrode]

@Coises said:

Would you try installing this version

With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU! Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?

[Coises]

@M-Andre-Z-Eckenrode said:
With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU!

Thank you. This error has been there since I first designed this regular expression search technique, in Columns++. It just apparently only shows up in rare circumstances. I’ve still never seen it.

Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?

Notepad++ uses an open source component called Scintilla to display editable text. Scintilla maintains the text in an internal “split buffer,” meaning that there can be a gap at an arbitrary position within the text. It uses that gap to be able to make changes that add or remove characters in the middle of the text without having to copy all the following text to a new location every time there’s one small change.

For a few reasons, when I implemented search, I wanted to use the same regular expression engine that Notepad++ uses, Boost.regex, but I wanted to include my own copy in the plugin and manage it directly, rather than going through the hybrid Scintilla/Boost interface that Notepad++ exposes, and I wanted to access the Scintilla split buffer directly. To do that, I had to create something called iterators.

In C++, an iterator is essentially an indirect way of referencing data that has simple operations like “move to the next character” and “tell me what character you’re pointed at.” I designed iterators for UTF-8, for single-byte character set ANSI and for double-byte character set ANSI that could traverse the Scintilla buffer, mind the gap, and return the results to the Boost.regex engine as if it were seeing a contiguous string of UTF-32 Unicode characters, regardless of the actual encoding.

Now, there’s a caveat when you ask Scintilla for the location of its internal buffer positions. You get two segments (because of the gap). But Scintilla warns that you can’t trust those positions once you do anything else in Scintilla, or once any user interaction is possible (even if the user doesn’t change the text). So I made sure to “invalidate” my internal pointers whenever control left my plugin: such as between finding text and changing it.

In order to ask Boost.regex for the replacement string when doing a change, I have to keep a particular Boost.regex structure that references the original match, so it knows how to interpret capture group substitutions. I was careful to invalidate the buffer pointers before I called Boost.regex.format. Except…

The match structure in Boost.regex doesn’t store the Scintilla character positions of the match. It stores iterators. And the iterators I designed, to be as efficient as possible, hold copies of the pointers into the Scintilla split buffer. So invalidating the pointers I use to generate iterators for Boost.regex does nothing to the iterators Boost.regex already stored in the match structure.

For reasons I can’t guess, both you and your brother seem to have something about your systems that causes Scintilla to reposition the gap in its internal buffer between find and replace. It never seems to happen here, but I finally realized that my attempt to protect against that happening is ineffective.

What I did to test was add a Scintilla call each time I invalidate the buffer pointers that causes Scintilla to move everything to the beginning and put the gap at the end. That means all the iterators will have the same pointers. It’s not a good long-term solution, though, because in a large file moving the gap to the end every single time anything changes could make response sluggish.

So it might take me a little while to decide on the best solution. I’m glad you documented the error so well. Now I know what to fix, I just have to work out the best way to do it.

[M Andre Z Eckenrode]

Thank you, @Coises, and best wishes for a long-term solution.

[Coises]

@M-Andre-Z-Eckenrode:

Search++ version 0.6.4 should also fix this, but in a way that won’t impact efficiency in very large files.

When you have a chance to check, please let me know if you find that it fails. I don’t think it will… but since I can’t reproduce the bug, I can’t be certain.

(I have not yet addressed the problem that caused Search++ to fail to load for your brother in Windows Sandbox and in virtual machines. I understand the cause, I just have to decide how to deal with it.)

Thank you so much for all your help!

[Alan Kilborn]

@Coises said:

…fail…in virtual machines

I’m somewhat curious about this, because, well, isn’t the idea of a VM that it runs the same as a real machine? If it doesn’t, isn’t it a lot less useful? OK, sure, it doesn’t run strictly the same entirely, but I’d say running an app, e.g. N++, either under a VM or not should produce the same results for the user.

I’ve run Notepad++ exclusively under a VM for 3+ years now, and have never noticed any behavior that was odd. I guess what I’m looking for is that you did something really “oddball” in your plugin, and that in general I have no reason to be concerned.

[M Andre Z Eckenrode]

@Coises said:

Search++ version 0.6.4 should also fix this

I’m happy to report that I’ve encountered no further anomalies during the course of several tests while using v0.6.4. You’re quite welcome, and thank YOU again.

[Coises]

@Alan-Kilborn said:
I’ve run Notepad++ exclusively under a VM for 3+ years now, and have never noticed any behavior that was odd. I guess what I’m looking for is that you did something really “oddball” in your plugin, and that in general I have no reason to be concerned.

Running in a VM or the Windows Sandbox was just an indirect cause. The real cause was the absence of a Visual C runtime redistributable library.

I reproduced the problem immediately in Windows 10 Sandbox, and I’m confident the cause in his brother’s virtual machines was the same. Most people have installed that runtime on a general purpose machine at some time or other as a side-effect of installing one application or another.

In a program with a proper installer, the installer would check for the presence of required libraries and start an installer for any missing libraries. (I’m sure you’ve seen that.) A Notepad++ plugin can’t really do that. Normally we just compile with everything statically linked.

The reason for the dependency is that I’m currently using pre-compiled versions of the ICU libraries. Those libraries, in turn, dynamically link to the Visual C runtime.

I think the proper solution to this must be to include the ICU source as a second project in my Visual Studio solution and compile everything with static linkage. I’m just not sure how to do that yet: I don’t think ICU has a drop-in project for a VS solution that uses MS Build, and as it happens, I know next to nothing about build systems. :-(

The next most proper would be to say something in the readme that if the plugin won’t load, it could be because you need to install these libraries.

Another possibility would be to use Windows’ version of ICU. That would mean being behind the latest version (Windows doesn’t routinely update its version of the libraries) and for certain not working on Windows versions earlier than Windows 10 version 1703. At present, Search++ doesn’t appear to work on Windows 7 anyway (I’m unsure about Windows 8/8.1); that appears also to be a result of using the ICU DLLs, and it might be resolvable if I compile them as part of the plugin; at present, I don’t know.

The simplest way would be to include the redistributable libraries in the zip file. At the least, that would technically be a GPL violation, since those libraries are not released under a GPL-compatible license. (As I understand it, you can depend on system libraries, but you can’t include a library in your distribution and claim it falls under the system library exception.) Aside from technicalities, I’m unsure if there might be other pitfalls to doing that.

[sevem47]

@Coises said:

Search++ doesn’t appear to work on Windows 7 anyway

Just for your information; this is something I can NOT confirm. For me Search++ even runs on my old, not up to date Windows 7 machine (see below debug info). Perhaps this is as well a question of having all the “correct libraries” to be able to run the plugin. If you want some further information, just tell me and I will see if I can provide you more details (but it is not a development machine; I just use it for writing documents).

Notepad++ v8.9.7   (32-bit)
Build time: Jul 14 2026 - 01:11:31
Scintilla/Lexilla included: 5.6.4/5.5.1
Boost Regex included: 1_90
pugixml included: 1.16
nlohmann JSON included: 3.12.0
Path: C:\ProgramData\UserApp\Notepad++\notepad++.exe
Command Line: 
Admin mode: OFF
Local Conf mode: OFF
Cloud Config: OFF
WinGUp: absent
disableNppAutoUpdate.xml: present
Periodic Backup: OFF
Placeholders: OFF
Scintilla Rendering Mode: SC_TECHNOLOGY_DEFAULT (0)
Multi-instance Mode: monoInst
asNotepad: OFF
File Status Auto-Detection: cdEnabledNew (for current file/tab only)
Dark Mode: OFF
Display Info:
    primary monitor: 1024x600, scaling 100%
    visible monitors count: 1
    installed Display Class adapters: 
        0000: Description - Intel(R) Graphics Media Accelerator 3150
        0000: DriverVersion - 8.14.10.2117
        0001: Description - Intel(R) Graphics Media Accelerator 3150
        0001: DriverVersion - 8.14.10.2117
OS Name: Windows 7 Starter (32-bit)
OS Build: 7601.0
Current ANSI codepage: 1252
Plugins: 
    ComparePlus (3)
    DSpellCheck (1.5)
    MarkdownViewerPlusPlus (0.8.2)
    mimeTools (3.1)
    NppConverter (4.7)
    NppExport (0.4)
    NPPJSONViewer (2.1.1)
    PoorMansTSqlFormatterNppPlugin (1.6.13.31502)
    Search++ (0.6.4)
    XMLTools (3.1.1.13)