Remove the external lexer support due to Scintilla 5

[Bas de Reuver]

Looks like Notepad++ is being updated to the latest Scintilla v5, and with that external lexers are no longer supported, is that correct?

This would mean that the CSV Lint plug-in no longer has syntax highlighting, and it shows this message at start-up at startup (thanks to @VinsWorldcom for mentioning).

As far as I can tell removing external lexers affects these plug-ins:

• CSV Lint
• GEDCOM Lexer @stan-mitchell
• Gmod Lua Lexer
• Lexamples
• Papyrus Script Lexer
• WLangLexer
• Lexer Example Project @Ekopalypse

This is a real shame, the syntax highlighting is a vital part of the CSV Lint plug-in ☹

Is there no way for Notepad++ to somehow keep supporting these external lexers going forward?

[Alan Kilborn]

@bas-de-reuver said in Remove the external lexer support due to Scintilla 5:

Is there no way for Notepad++ to somehow keep supporting these external lexers going forward?

I am lending my voice to the desire to keep external lexer support…somehow.

[PeterJones]

@Bas-de-Reuver ,

I don’t know the answer to this, but I wonder if it’s really a matter of the new Scintilla doesn’t accept the old style of external lexers, but if someone were to make an iLexer5-based plugin, would that work?

One of the people more intimately involved with the Scintilla 5 migration (maybe @chcg , since he did the first Scintilla 5 commit in NPP) might be able to comment more intelligently on this topic.

[PeterJones]

Further to what I said,

The “Lexilla” section of https://www.scintilla.org/ScintillaDoc.html says,

Lexer libraries that provide the same functions as Lexilla may provide lexers for use by Scintilla, augmenting or replacing those provided by Lexilla.

… which strongly implies that external libraries can also supply lexers to Scintilla.

And the first full paragraph in https://www.scintilla.org/LexillaDoc.html says that Lexilla itself can be either statically linked or dynamically linked. So I think there must be a way to make the new Scintilla accept an external Lexer… in which case, a NPP Plugin should be able to define an external Lexer.

Unfortunately, I don’t have the skills (or the time to develop the skills) to try to write an iLexer5-based lexer, plugin or otherwise.

[Bas de Reuver]

@peterjones Good find, that looks hopefull at least. If that’s possible, then obviously I’ll update the plug-in asap so that it works with the new Scintilla version/Notepad++ version.

For now I guess I’ll have to wait(?) for the next Notepad++ update to be finalised. And then the best thing to do, is I’ll first try to update the NppPluginLexerExample project and then implement the changes also in the CSV Lint plugin.

[mpheath]

@peterjones said in Remove the external lexer support due to Scintilla 5:

…
Unfortunately, I don’t have the skills (or the time to develop the skills) to try to write an iLexer5-based lexer, plugin or otherwise.

Try compiling the example:

https://github.com/ScintillaOrg/lexilla/blob/master/examples/SimpleLexer/SimpleLexer.cxx

This how it looks in SciTE using SimpleLexer.dll to do the lexing.

[Michael Vincent]

@bas-de-reuver said in Remove the external lexer support due to Scintilla 5:

Is there no way for Notepad++ to somehow keep supporting these external lexers going forward?

I may be way off base here, but I don’t think this is the way to go. I think you need to use Lexilla. This is how Scintilla 5+ handles it. I think they just removed the lexing capability from Scintilla itself.

Cheers.

[Michael Vincent]

@peterjones said in Remove the external lexer support due to Scintilla 5:

but if someone were to make an iLexer5-based plugin, would that work

@Bas-de-Reuver

Perhaps, there is a Scintilla call (SCI_SETILEXER) to set the Lexer.

Cheers.

[rdipardo]

@mpheath said in Remove the external lexer support due to Scintilla 5:

Try compiling the example:

https://github.com/ScintillaOrg/lexilla/blob/master/examples/SimpleLexer/SimpleLexer.cxx

Notepad++ compiles lexer properties right into the application, so the external lexer’s module path would have to be known at build time :(

SciTE is more configurable; it loads properties at runtime, which the demo takes for granted:

/* It can be demonstrated in SciTE like this, substituting the actual shared library location as lexilla.path:
lexilla.path=.;C:\u\hg\lexilla\examples\SimpleLexer\SimpleLexer.dll
lexer.*.xx=simple
style.simple.1=fore:#FF0000
*/

Notepad++ opted for static Scintilla linkage in version v8.

But now that Lexilla is a standalone library, maybe plugin authors could start rolling their own lexilla.dll and ship that . . . ?

[rdipardo]

maybe plugin authors could start rolling their own lexilla.dll [. . .] ?

Nope: they went the static route with Lexilla, too.

You could try upvoting this upstream issue; but it won’t help until N++ can load properties at runtime the way SciTE does.

[mpheath]

@rdipardo

I am not sure if statically linked is the problem. The image I posted previously is the Sc1 version of SciTE (statically linked with Scintilla and Lexilla). If Sc1 can do it, so I would expect also for Notepad++.

The upstream issue you mention is about Scintillua with Sc1. Basically wants an LPeg lexer priority before Lexilla lexer. If Neil knows the patch is good, it would have been committed. I do not use Scintillua and the other poster has not returned a response to confirm if the patch is OK. Not sure if this issue is related to Notepad++ and it’s external lexing as Notepad++ does not use Scintillua.

[Ekopalypse]

Scintilla still supports external lexers, but it won’t work without support from Npp.
I have asked Npp’s position on this in the commit.

[rdipardo]

@mpheath

The image I posted previously is the Sc1 version of SciTE (statically linked with Scintilla and Lexilla).

After a bit of grokking it works for me, too (*). I’ve automated the steps for anyone who wants to try it.

You’ll need git, mercurial and CMake all in your PATH, and at least Visual Studio 2017 with the C++ workload installed.

Start the “x64 Native Tools” or “x86_x64 Cross Tools” Developer Command Prompt and run:

git clone --recursive -b scite-gtk3 https://bitbucket.org/rdipardo/lexilla-dev.git
cd lexilla-dev\demo
.\sc1-with-ext-lexer.bat

---

(*) I usually let MsBuild choose the default platform of x86, but a 32-bit DLL will export decorated functions that won’t match the definitions in Lexilla.h, and so fail to load.

[rdipardo]

@Bas-de-Reuver, @Ekopalypse,

Remove Replace the external lexer support due to Scintilla 5 with the new Lexilla interface

https://github.com/notepad-plus-plus/notepad-plus-plus/pull/11468

[Stan Mitchell130]

@bas-de-reuver
I just read your post. As an author of an external lexer plugin, I too am concerned about the impact of removing external lexer support. When I did my last release of GEDCOM lexer 3 years ago, I thought about submitting the lexer to Scintilla so it would become an “internal” lexer. Maybe that is a path forward if all else fails.

[mpheath]

@stan-mitchell130

In case you missed it… @rdipardo posted with a link showing external lexer restoration:

Restore external lexer library support #11468

and has been merged recently into the repository with the message:

Make external language library work again after upgrading to Scintilla5

Make external language library work again after upgrading to Scintilla5
Make external lexer library work again after upgrading to Scintilla5.
Old external lexer libraries needs to add CreateLexer export function which returns ILexer5 instance (Lexilla protocol interface of Scintilla5).
Tested with papyrus lexer plugin, this external lexer plugin is compatible with Notepad++ next release:
https://github.com/blu3mania/npp-papyrus

External lexers were removed temporarily. Now, have been added back in as ILexer5 compatible!

[Stan Mitchell130]

@mpheath
Thanks for the update! I’ll have to dig into the ILexer5 changes.

[Bas de Reuver]

I was looking at this example, and I added a method called CreateLexer to the file UnmanagedExports.cs file of the C# LexerExample.

With this added, the plugin works again, there is no error message (“Loading CreateLexer function failed. Edifactlexed.dll is not compatible with the current version of Notepad++.”) anymore at startup of Notepad++, and the syntax highlightng also still works. 😃

@Ekopalypse Do I understand correctly that this is the only thing that needed to be added, I mean as far as the interface goes there are no changes between iLexer4 and iLexer5?

[Ekopalypse]

@bas-de-reuver

No, there are 3 additional methods defined by ILexer5, but it looks like they are not used yet.
However, this could become an issue in the future.

class ILexer5 : public ILexer4 {
public:
        virtual const char * SCI_METHOD GetName() = 0;
        virtual int SCI_METHOD  GetIdentifier() = 0;
        virtual const char * SCI_METHOD PropertyGet(const char *key) = 0;
};

[Shridhar Kumar]

@Ekopalypse, thanks for the hints.

Without the implementation of the 3 additional methods, Notepad++ will crash when any of these following functions are called from the PythonScript console (or, potentially its SCI_xxx equivalents are invoked by another plugin installed in Notepad++):

editor.getLexer()
editor.getLexerLanguage()
editor.setProperty('TESTING', 'Hello, World!')
editor.getProperty('TESTING')

See issue: https://github.com/BdR76/CSVLint/issues/26
Fixed in PR: https://github.com/BdR76/CSVLint/pull/27