NPP can't handle large files

Herriman Coder

At work I frequently handle large-ish files. Example: A text file with 70K lines of XML and 4.16 MB in size. Admittedly this is an edge case for most, since NPP is prolly mostly used for source code editing. But NPP really flops hard when trying to SAVE this type of file. Conversely, PilotEdit Lite handles the file fast and no problem. But I don’t want to use PilotEdit - NPP is way better; it just can’t handle large-ish files.

First, I don’t consider a 4 Mb text file a huge file. NPP should be able to handle that. It’s not like we’re talking about a 200 Gb file here.

But on to my use case:

Download a ton of xml metadata from Business Central using Postman
Copy/paste that into NPP
Try to save the file
Give up and kill NPP after 2 minutes of unresponsiveness.

YET when I do the same thing in PilotEdit, it saves in 2 seconds. Done.

Why can’t NPP do that?

I can then open that file in NPP, takes about 5 seconds, no worries there. But if I try to change and save anything, again, major choking and struggle.

Can you give some love to handing this type of file? There must be some little bit of magic sauce in PilotEdit that enables easy handling of a 4 mb text file – can the NPP gurus discover the sauce and code it in?

I would be happy to send you the file - it’s not too sensitive, but I don’t want to post it online because it exposes the metadata of BC specific to one of our customers.

Alan Kilborn

@Nate-Jensen

Have you tried adjusting anything here?:

PeterJones

@Nate-Jensen ,

4MB is not a “large” file by most definitions – even Notepad++'s own “large file” definition defaults to 200MB, as Alan’s screenshot showed.

My quick experiments are that it works fast with 4MB XML (I took my shortcuts.xml file, and used Ctrl+A/Ctrl+V to double it in size until it’s 4Mb, then saved it). But then I tried it with Function List panel turned on (View > Function List), and that slowed it down so that adding and deleting a space was fine, but hitting Save after that non-change took about 30sec. (The “function list” definition for XML shows essentially any tag with attributes, which seems a bit much to me.) So in addition to playing with the settings Alan showed, if you have Function List shown, turn it off when you’re dealing with XML with thousands of tags.

(I actually tried an XML FunctionList definition that only shows tags without attributes instead of tags with attributes, and, at least for my dummy file, since there were many fewer of those, it worked a lot faster, even with >50k lines and 4MB; but that may be an artifact of the Notepad++ config file format using fewer non-attributed tags than attributed tags, rather than something that’s necessarily true or many or most XML files)

Also, if you have one or more XML-related Plugins (like XML Tools), it may be that their extra processing (especially if you have automatic syntax checking / XML validation turned on) is slowing down Notepad++'s performance; you might want to disable those plugins (or at least any automatic behavior), and see if it gets faster.

Coises

The problem is specifically saving, right… nothing else is unreasonably slow?

Is there any chance that anti-malware is involved? Just as an experiment, try disabling anti-malware before you save.

Also, try running notepad++ -noPlugin to be sure a plugin isn’t doing something when you try to save. (Check to make sure your plugins really didn’t load… as small a thing as typing -noplugin instead of -noPlugin will cause the parameter to be ignored.)

If neither of those things makes a difference, I guess I’d next look at Windows Task Manager and/or Resource Monitor (Task Manager | Performance | Open Resource Monitor at the lower left) to try to figure out what’s happening — is CPU or IO running at max? Is some other application you didn’t expect becoming active when you save?

Michael Vincent

@Nate-Jensen said in NPP can’t handle large files:

70K lines of XML

XML files you say … are you using the “Function List” panel? If so, close it and watch performance improve.

Reference: https://github.com/notepad-plus-plus/notepad-plus-plus/issues/8796#issuecomment-685739437

Cheers.

Herriman Coder

I found the culprit:

XML Tools -> Enable XML syntax auto-check absolutely CRIPPLES performance on any XML file of non-trivial size. When I turned that off, my 4 mb xml file now saves changes instantly.

Thanks for all the comments and suggestions - those led me to look down this road.

Suggestion for NPP core product: Detect misbehaving extensions. If an extension causes NPP to totally lock up for more than 10 seconds or so (in my case it was over 3 minutes), it should pop up a message that says “It appears that XML Tools (or whatever extension) has become unresponsive. Options: Kill the extension process; Let it run / Don’t ask again / ETC”

Thanks everyone!!

Herriman Coder

Side note: I forgot to mention that Function List was already turned off for me, so that wasn’t a factor. But when I turned it on, I saw a ton of slowness. So I turned it back off. After turning that off, I still had all the original slowness during save, which led me to the XML Tools problem as noted in my previous post.

rdipardo

@Herriman-Coder

Suggestion for NPP core product: Detect misbehaving extensions.

I’m guessing that “extensions” means plugins here, in which case you would want to ping the plugin’s maintainer: https://github.com/morbac/xmltools/issues

With the exception of the core developer’s own plugins, there is absolutely no co-ordination between the “core product” and the hundreds of independently maintained plugins out there.

If instead you’re talking about file extensions, see https://github.com/notepad-plus-plus/notepad-plus-plus/issues/11319

Mark Olson

@rdipardo
Does NPP track what plugins receive notifications?

Because if so, it might be possible (though by no means easy) to log each plugin’s response times to notifications. This would probably have to be disabled by default to reduce overhead, but it seems potentially do-able.

My rough sketch would be something like:

Check time when notification was sent to plugin.
Check time when plugin responded
If the plugin’s response time was slow (say, greater than 1 second), add to a log file.
On startup, check the log file, and if it’s non-empty, put up a messagebox saying something like XMLTools plugin took 2.5 seconds to respond to the NPPM_FILEBEFORESAVE notification. Disable this plugin?

Coises

@Mark-Olson said in NPP can’t handle large files:

Does NPP track what plugins receive notifications?

Notepad++ calls each plugin in turn to relay notifications. It does the call in a try/catch block so it can catch exceptions. (Unfortunately, that doesn’t always work, as I discovered when I had an exception inside the dialog procedure for a modal dialog — Windows can fall down and go boom without ever raising an exception C++ can catch.)

My guess is that the way to implement this would be to set a timer just before making the call, so there would be an interrupt if the process ran too long. How long is too long, though, would be very dependent on what one expected the plugin to do. And whether it’s possible to interrupt the GUI thread in that way in Windows is something I haven’t yet had cause to learn. If possible, though, I’d expect the same sort of thing you get from a Javascript routine running too long in a browser: a message box much like @Herriman-Coder suggested, asking whether to continue, stop (this time) or disable the plugin.

rdipardo

@Coises

My guess is that the way to implement this would be to set a timer just before making the call, so there would be an interrupt if the process ran too long.

SendMessageTimeout with the SMTO_NORMAL flag would achieve something like that: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendmessagetimeoutw

As you point out, however, that would entail picking a fixed, arbitrary number of milliseconds for every single plugin message that Npp would dispatch.

A more provident implementation would have all plugins running on separate threads, the way Sublime Text plugins have done since version 3.

I think the OP was really asking for some way to detect the source of a (potentially problematic) message; e.g., if XMLTools wants to check syntax, ask the user for permission first. Of course that’s impossible, and any workaround would penalize every other plugin at the same time.