@M-Andre-Z-Eckenrode said:
With v0.6.3.2 in place, I re-ran several of the tests I’ve been doing, and found no anomalies this time, so that seems to have taken care of it. Thank YOU!
Thank you. This error has been there since I first designed this regular expression search technique, in Columns++. It just apparently only shows up in rare circumstances. I’ve still never seen it.
Out of curiosity, is there any kind of basic explanation you can relay in layman’s terms for why it was doing what it was doing?
Notepad++ uses an open source component called Scintilla to display editable text. Scintilla maintains the text in an internal “split buffer,” meaning that there can be a gap at an arbitrary position within the text. It uses that gap to be able to make changes that add or remove characters in the middle of the text without having to copy all the following text to a new location every time there’s one small change.
For a few reasons, when I implemented search, I wanted to use the same regular expression engine that Notepad++ uses, Boost.regex, but I wanted to include my own copy in the plugin and manage it directly, rather than going through the hybrid Scintilla/Boost interface that Notepad++ exposes, and I wanted to access the Scintilla split buffer directly. To do that, I had to create something called iterators.
In C++, an iterator is essentially an indirect way of referencing data that has simple operations like “move to the next character” and “tell me what character you’re pointed at.” I designed iterators for UTF-8, for single-byte character set ANSI and for double-byte character set ANSI that could traverse the Scintilla buffer, mind the gap, and return the results to the Boost.regex engine as if it were seeing a contiguous string of UTF-32 Unicode characters, regardless of the actual encoding.
Now, there’s a caveat when you ask Scintilla for the location of its internal buffer positions. You get two segments (because of the gap). But Scintilla warns that you can’t trust those positions once you do anything else in Scintilla, or once any user interaction is possible (even if the user doesn’t change the text). So I made sure to “invalidate” my internal pointers whenever control left my plugin: such as between finding text and changing it.
In order to ask Boost.regex for the replacement string when doing a change, I have to keep a particular Boost.regex structure that references the original match, so it knows how to interpret capture group substitutions. I was careful to invalidate the buffer pointers before I called Boost.regex.format. Except…
The match structure in Boost.regex doesn’t store the Scintilla character positions of the match. It stores iterators. And the iterators I designed, to be as efficient as possible, hold copies of the pointers into the Scintilla split buffer. So invalidating the pointers I use to generate iterators for Boost.regex does nothing to the iterators Boost.regex already stored in the match structure.
For reasons I can’t guess, both you and your brother seem to have something about your systems that causes Scintilla to reposition the gap in its internal buffer between find and replace. It never seems to happen here, but I finally realized that my attempt to protect against that happening is ineffective.
What I did to test was add a Scintilla call each time I invalidate the buffer pointers that causes Scintilla to move everything to the beginning and put the gap at the end. That means all the iterators will have the same pointers. It’s not a good long-term solution, though, because in a large file moving the gap to the end every single time anything changes could make response sluggish.
So it might take me a little while to decide on the best solution. I’m glad you documented the error so well. Now I know what to fix, I just have to work out the best way to do it.
