Remove the external lexer support due to Scintilla 5

[mpheath]

@peterjones said in Remove the external lexer support due to Scintilla 5:

…
Unfortunately, I don’t have the skills (or the time to develop the skills) to try to write an iLexer5-based lexer, plugin or otherwise.

Try compiling the example:

https://github.com/ScintillaOrg/lexilla/blob/master/examples/SimpleLexer/SimpleLexer.cxx

This how it looks in SciTE using SimpleLexer.dll to do the lexing.

[Michael Vincent]

@bas-de-reuver said in Remove the external lexer support due to Scintilla 5:

Is there no way for Notepad++ to somehow keep supporting these external lexers going forward?

I may be way off base here, but I don’t think this is the way to go. I think you need to use Lexilla. This is how Scintilla 5+ handles it. I think they just removed the lexing capability from Scintilla itself.

Cheers.

[Michael Vincent]

@peterjones said in Remove the external lexer support due to Scintilla 5:

but if someone were to make an iLexer5-based plugin, would that work

@Bas-de-Reuver

Perhaps, there is a Scintilla call (SCI_SETILEXER) to set the Lexer.

Cheers.

[rdipardo]

@mpheath said in Remove the external lexer support due to Scintilla 5:

Try compiling the example:

https://github.com/ScintillaOrg/lexilla/blob/master/examples/SimpleLexer/SimpleLexer.cxx

Notepad++ compiles lexer properties right into the application, so the external lexer’s module path would have to be known at build time :(

SciTE is more configurable; it loads properties at runtime, which the demo takes for granted:

/* It can be demonstrated in SciTE like this, substituting the actual shared library location as lexilla.path:
lexilla.path=.;C:\u\hg\lexilla\examples\SimpleLexer\SimpleLexer.dll
lexer.*.xx=simple
style.simple.1=fore:#FF0000
*/

Notepad++ opted for static Scintilla linkage in version v8.

But now that Lexilla is a standalone library, maybe plugin authors could start rolling their own lexilla.dll and ship that . . . ?

[rdipardo]

maybe plugin authors could start rolling their own lexilla.dll [. . .] ?

Nope: they went the static route with Lexilla, too.

You could try upvoting this upstream issue; but it won’t help until N++ can load properties at runtime the way SciTE does.

[mpheath]

@rdipardo

I am not sure if statically linked is the problem. The image I posted previously is the Sc1 version of SciTE (statically linked with Scintilla and Lexilla). If Sc1 can do it, so I would expect also for Notepad++.

The upstream issue you mention is about Scintillua with Sc1. Basically wants an LPeg lexer priority before Lexilla lexer. If Neil knows the patch is good, it would have been committed. I do not use Scintillua and the other poster has not returned a response to confirm if the patch is OK. Not sure if this issue is related to Notepad++ and it’s external lexing as Notepad++ does not use Scintillua.

[Ekopalypse]

Scintilla still supports external lexers, but it won’t work without support from Npp.
I have asked Npp’s position on this in the commit.

[rdipardo]

@mpheath

The image I posted previously is the Sc1 version of SciTE (statically linked with Scintilla and Lexilla).

After a bit of grokking it works for me, too (*). I’ve automated the steps for anyone who wants to try it.

You’ll need git, mercurial and CMake all in your PATH, and at least Visual Studio 2017 with the C++ workload installed.

Start the “x64 Native Tools” or “x86_x64 Cross Tools” Developer Command Prompt and run:

git clone --recursive -b scite-gtk3 https://bitbucket.org/rdipardo/lexilla-dev.git
cd lexilla-dev\demo
.\sc1-with-ext-lexer.bat

---

(*) I usually let MsBuild choose the default platform of x86, but a 32-bit DLL will export decorated functions that won’t match the definitions in Lexilla.h, and so fail to load.

[rdipardo]

@Bas-de-Reuver, @Ekopalypse,

Remove Replace the external lexer support due to Scintilla 5 with the new Lexilla interface

https://github.com/notepad-plus-plus/notepad-plus-plus/pull/11468

[Stan Mitchell130]

@bas-de-reuver
I just read your post. As an author of an external lexer plugin, I too am concerned about the impact of removing external lexer support. When I did my last release of GEDCOM lexer 3 years ago, I thought about submitting the lexer to Scintilla so it would become an “internal” lexer. Maybe that is a path forward if all else fails.

[mpheath]

@stan-mitchell130

In case you missed it… @rdipardo posted with a link showing external lexer restoration:

Restore external lexer library support #11468

and has been merged recently into the repository with the message:

Make external language library work again after upgrading to Scintilla5

Make external language library work again after upgrading to Scintilla5
Make external lexer library work again after upgrading to Scintilla5.
Old external lexer libraries needs to add CreateLexer export function which returns ILexer5 instance (Lexilla protocol interface of Scintilla5).
Tested with papyrus lexer plugin, this external lexer plugin is compatible with Notepad++ next release:
https://github.com/blu3mania/npp-papyrus

External lexers were removed temporarily. Now, have been added back in as ILexer5 compatible!

[Stan Mitchell130]

@mpheath
Thanks for the update! I’ll have to dig into the ILexer5 changes.

[Bas de Reuver]

I was looking at this example, and I added a method called CreateLexer to the file UnmanagedExports.cs file of the C# LexerExample.

With this added, the plugin works again, there is no error message (“Loading CreateLexer function failed. Edifactlexed.dll is not compatible with the current version of Notepad++.”) anymore at startup of Notepad++, and the syntax highlightng also still works. 😃

@Ekopalypse Do I understand correctly that this is the only thing that needed to be added, I mean as far as the interface goes there are no changes between iLexer4 and iLexer5?

[Ekopalypse]

@bas-de-reuver

No, there are 3 additional methods defined by ILexer5, but it looks like they are not used yet.
However, this could become an issue in the future.

class ILexer5 : public ILexer4 {
public:
        virtual const char * SCI_METHOD GetName() = 0;
        virtual int SCI_METHOD  GetIdentifier() = 0;
        virtual const char * SCI_METHOD PropertyGet(const char *key) = 0;
};

[Shridhar Kumar]

@Ekopalypse, thanks for the hints.

Without the implementation of the 3 additional methods, Notepad++ will crash when any of these following functions are called from the PythonScript console (or, potentially its SCI_xxx equivalents are invoked by another plugin installed in Notepad++):

editor.getLexer()
editor.getLexerLanguage()
editor.setProperty('TESTING', 'Hello, World!')
editor.getProperty('TESTING')

See issue: https://github.com/BdR76/CSVLint/issues/26
Fixed in PR: https://github.com/BdR76/CSVLint/pull/27

[rdipardo]

@shridhar-kumar,
even a basic implementation of Scintilla::ILexer5 should implement every method you see here. @ozone10’s rainlexer is a good, complete example.

[rdipardo]

Fixed in PR: https://github.com/BdR76/CSVLint/pull/27

Builds and works fine. I left comments on some .NET-related stuff, but here’s a diff to sum it all up:

diff --git a/CSVLintNppPlugin/PluginInfrastructure/Lexer.cs b/CSVLintNppPlugin/PluginInfrastructure/Lexer.cs
index f636ee7..44fcee4 100644
--- a/CSVLintNppPlugin/PluginInfrastructure/Lexer.cs
+++ b/CSVLintNppPlugin/PluginInfrastructure/Lexer.cs
@@ -11,7 +11,7 @@ internal static class ILexer
     {
         public static readonly string Name = "CSVLint\0";
         public static readonly string StatusText = "CSV Linter and validator\0";
-        static readonly int LexerID = 1976;
+        public static readonly int LexerID = 1976;
 
         public static char separatorChar = ';';
         public static List<int> fixedWidths;
@@ -341,7 +341,8 @@ public static int Version(IntPtr instance)
              * ILexer5 must be provided for Scintilla version 5.0 or later.
              */
             //GC.Collect();  // test to see if the methods do get garbage collected
-            return 2;
+            // https://sourceforge.net/p/scintilla/code/ci/default/tree/include/ILexer.h#l45
+            return 3; 
         }
 
         // virtual void SCI_METHOD Release() = 0
@@ -822,14 +823,14 @@ public static int SubStylesLength(IntPtr instance, int style_base)
         public static int StyleFromSubStyle(IntPtr instance, int sub_style)
         {
             // For a sub style, return the base style, else return the argument.
-            return 0;
+            return sub_style;
         }
 
         // virtual int SCI_METHOD PrimaryStyleFromStyle(int style) = 0;
         public static int PrimaryStyleFromStyle(IntPtr instance, int style)
         {
             // For a secondary style, return the primary style, else return the argument.
-            return 0;
+            return style;
         }
 
         // virtual void SCI_METHOD FreeSubStyles() = 0;
@@ -886,7 +887,7 @@ public static IntPtr PropertyGet(IntPtr instance, IntPtr key)
             /*  returns Property Value.
              *  Equivalent to editor.getProperty('key') in PythonScript.
              */
-            return Marshal.StringToHGlobalAnsi("N/A");
+            return IntPtr.Zero;
         }
     }
 }

[Ekopalypse]

@shridhar-kumar said in Remove the external lexer support due to Scintilla 5:

editor.setProperty(‘TESTING’, ‘Hello, World!’)

Yes, that has changed in the meantime.
But are you using the latest PythonScript version?
It doesn’t crash on me, but there seems to be an issue with the property calls.

[Alan Kilborn]

@ekopalypse said in Remove the external lexer support due to Scintilla 5:

But are you using the latest PythonScript version?

You should probably state what that is.
Was it the intent of your Debug info window to show it? – it doesn’t.

[Alan Kilborn]

Here’s how I see the bug, on N++ 8.4.2(RC2):