would it be possible to use JNI to provide some Java functionality to a C++ plugin?
AFAIK this would certainly be possible assuming you know enough about JNI.
do i need to do any character encoding/decoding
Most likely. Getting the text from Scintilla just gives you a const char * to the text however it is currently set to be encoded. You’d also want to make sure any text inserted/added to Scintilla also has the proper encoding (SCI_GETCODEPAGE may be useful to you). Depending on what you are doing you can probably do this in either Java or C++ (I don’t know enough about JNI). When writing C++ plugins I’ve normally used MultiByteToWideChar() and WideCharToMultiByte().