Big files cannot opened in v8.6.4 (x64)

PeterJones

@xomx said in Big files cannot opened in v8.6.4 (x84):

What is the “.mrk” file for?

… and even more, @Klaus-Lehmann , do you have a built-in syntax highlighter or UDL syntax highlighter active for .mrk files? if so, you might want to use Settings > Preferences > Performance to disable syntax highlighting on large files, because >2GB may be too big for scintilla/lexilla to highlight.

mkupper

@Klaus-Lehmann I tested and verified that npp v8.6.4 (x64) works with 3gb files (3,221,225,472 byte) and 16gb (17,179,869,184 byte) files.

My test was using files that contained lines with 128 characters including the CRLF per line:

123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456
123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456
123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456
123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456789.123456
...

I used the .txt file extension.

Things to try:

Rename your zdb.mrk to zdb.txt and see if that opens.
Is your f:\tmp on a file or NAS server? That may have issues with large files. Try copying zdb.mrk to drive C: or other local disk drive that has space.
A bit harder to track down is if there is something about the .mrk file content that bothers Notepad++ or Scintilla.

Coises

As a test, would you download Single file 64-bit executable called Sc1 from SciTE and try to open your file?

SciTE is a demonstration application for the Scintilla edit control. Notepad++ depends on Scintilla, but adds a lot of features around it. If SciTE fails on your file in the same way, we can conclude that either Scintilla or the file itself is the problem. If SciTE can load it, then the problem must have something to do with Notepad++.

Klaus Lehmann

@PeterJones
thanks for the good question.
of course I don’t use “a built-in syntax highlighter or UDL syntax highlighter” active.
one of the worst was the xml-syntax-highliter, if I would use them in BIG xml-files ;-)
Yours klaus

Klaus Lehmann

@Coises
interesting. I have had not any knowledge about this tool “Sc1”
please look at the picture:

is this true, that there exists a 2GB border?
it means really no file has to be bigger than 2 GB?

bad moment: I can’t remember, use bigger than 2GB-text-files.
I f this true, than I know only one texteditor, CRISP. Her has no problem, to open bigger than 2 GB.
Before 5-10 years, I was a mainyl daily user with Crisp. but before 5-10 years I changed completely onto Notepad++ (and some years later onto 64bit-version).

Thanks, klaus

Klaus Lehmann

thanks You very nuch for Your test. this means for me, the problem is on my side. not on side from notepad++
strange.

Your advises like:
copy *.mrk onto *,txt. result: same thing.
*.mrk is a normal ascii-based file. every 100 charcters or 1000 chars there is a full \x0d\x0a (CRLF).
normally I do my work on local drives. and just know also on a local disc (ssd).

thanks Klaus

Klaus Lehmann

@xomx
“What is the “.mrk” file for?”

I forgot to answer: *.mrk is a normal extension for us, who are librarians.

if there is someone interested to bekome mor infos, please look at:
[http://loc.gov](Link Adresse) -> better in german here: https://www.alma-dach.org/alma-marc/
and than in english: [https://www.loc.gov/marc/](Link Adresse) (but its strong stuff, not for everyone ;-)

mark exist in 3 forms:
mrk, ist more human readable, mainly ascii.
mrc, is absolute binary. it has a veyr logic structure (but I have spend some years to understand it) ;-)
and than my “loveliest”: marc-xml (I hate xml!). 80-90% useless information. the rest is usefull (but, attention: its my personal opinion. bloody programmers loves xml ;-)

and If “we” use them, there can exist a mrk-file as a big data file:
my mrk ist 3 GB. as a mrc it is a little bit more: 3,3GB
as an xml? it is 11GB

and what is the “zdb”? ;-)
“Zeitschriften Datenbank”. worlds biggest catalogue with existing and old periodicals in libraries.
content: from 2024-03-15 12:25 655M 2.113.828 titels (ZDB-Titel Gesamtbestand, Format MARC 21 (Stand: 12.03.2024 01:30)
but this is stuff for librarians ;-)

Yours Klaus

Coises

@Klaus-Lehmann said in Big files cannot opened in v8.6.4 (x84):

is this true, that there exists a 2GB border?
it means really no file has to be bigger than 2 GB?

The message is not worded very well. It only means that the program is configured to warn you when you try to open a file larger than 2 GB. If you answer “Ja” it will proceed to open the file.

I checked, before posting that suggestion, that it can in fact open a very large file (mine was over 4 GB).

Coises

@Klaus-Lehmann said in Big files cannot opened in v8.6.4 (x84):

thanks You very nuch for Your test. this means for me, the problem is on my side. not on side from notepad++

I’m not sure to whom you are responding here, so my apologies if this doesn’t make any sense.

Have you checked your system’s error logs to see if there is a hardware error reading the file? I’m not sure if this might have changed in Windows 11, but in Windows 10 you right-click the Windows Start icon, then select Event Viewer, then expand Windows Logs and look at the Application and (most importantly) System logs. Once you select one of those, you can use Filter current log… in the right-hand pane to choose to see only Critical and Error messages; but most useful would be to recreate the failure at a known time first, so you will know at which time in the logs to look.

xomx

https://librarycarpentry.org/lc-marcedit/instructor/03-working-with-MARC-files.html

A sample data for testing:
https://github.com/LibraryCarpentry/lc-marcedit/blob/main/episodes/data/marc_sample_data.zip

I made a 3.5GB mrk-file from the included marc_sample_data.mrk (just by multiplying its content) - N++ loaded that ok.

I also tried to open that small included mrc-file there and it does not contain any EOL in it. So if someone tries to open such a mrc-file larger than 2GB, the exact exception as reported will occur, as the Scintilla currently cannot handle lines longer than the INT_MAX chars.

@Klaus-Lehmann
Could you share your problematic mrk-file for testing?

Coises

@xomx said in Big files cannot opened in v8.6.4 (x84):

Scintilla currently cannot handle lines longer than the INT_MAX chars.

I can’t find documentation for this, but it appears to be correct. I constructed some large files made of nothing but the character “a” repeated enough times to make 1GiB and 2GiB files, then tried to open them with SciTE and with Notepad++. The 1GiB file was very slow to open, and unusable even for viewing, in both programs; the 2GiB file just crashed with no message at all.

It appears that no Scintilla-based editor is a good choice for large files that are not line-oriented, and hence may have very long stretches with no x0A or x0C characters in them. I don’t know what would be a good editor or viewer for such files.

Klaus Lehmann

@xomx
sorry for small delay
here’s Your link: [http://allegronet.de/zdb.zip](Link Adresse) .
527MB as zip. content: a pure mrk-file. unzip it. content than: 3 GB.
content is public data. allegronet is my server. please don’t go first onto http://allegronet.de. it will not work. try please only with the exact link: [http://allegronet.de/zdb.zip](Link Adresse) . thanks.

xomx

@Klaus-Lehmann
I can open this file ok, used the same version as yours for the test:

Notepad++ v8.6.4   (64-bit)
Build time : Feb 20 2024 - 00:12:59
Path : C:\Program Files\Notepad++\notepad++.exe
Command Line : C:\TEMP\NPP\zdb.mrk
Admin mode : OFF
Local Conf mode : OFF
Cloud Config : OFF
OS Name : Windows 10 Enterprise LTSC 2021 (64-bit)
OS Version : 21H2
OS Build : 19044.4170
Current ANSI codepage : 1250
Plugins : 
    mimeTools (3.1)
    NppConverter (4.6)
    NppExport (0.4)

Will try your build of Win11 later.

xomx

@Klaus-Lehmann
I successfully opened it also on the Win11 22H2 (I do not have your 23H2 available).

So it is something specific on your PC. We can eliminate the possible problem with a N++ setting by downloading the portable version equivalent, which does not need to be installed, but just unzipped into a folder and then run the notepad++.exe from there. Then, if you are able to open that zdb.mrk in your portable N++, the problem lies with your N++ installation. Otherwise the problem is probably outside the N++ control (a Windows setting, installed app or shell extension, AV, storage place problem, etc…).

Klaus Lehmann

@xomx
sorry for me, that my answer arrived You very late. but there was too much jobs with family ;-)
ok. I think, I will see my problem and can explain it.
1st: thanks to @Coises , who gave me hint with Sc1.
Yes! with Sc1 I can open without problems m ymrk-file with 3GB content.
This talks to me, the propblem is based on my machine. but where?
some days have gone
deInstall and install N++ at new. and very certain: on another machine (why? it depends from appdata\roaming\notepadd++ . there resides config.xml

I’m shure, here was my problem:

old (not working since 2 weeks WITH 3GB file) config.xml:
<GUIConfig name="largeFileRestriction" fileSizeMB="4096" isEnabled="no" allowAutoCompletion="no" allowBraceMatch="no" allowSmartHilite="no" allowClickableLink="no" deactivateWordWrap="yes" suppress2GBWarning="yes" />

new (working) config.xml
<GUIConfig name=“largeFileRestriction” fileSizeMB=“200” isEnabled=“yes” allowAutoCompletion=“no” allowBraceMatch=“no” allowSmartHilite=“no” allowClickableLink=“no” deactivateWordWrap=“yes” suppress2GBWarning=“yes” />

I think last line brings me good results. all works.
again: last line was my solution. Yes I know, there are some "misunderstoodings":
isEnabled="yes"   and  fileSizeMB="200" . very unlogical. but it works!

Thanks for all, who has helped me.
Yours Klaus

xomx

@Klaus-Lehmann
Congrats, you’ve just uncovered a N++ misconfiguration bug (as you have already found with the Sc1.exe test - this is not a Scintilla issue but a N++ one):

I’ll try to fix it but no ETA.

xomx

fixed, will be in the upcoming v8.6.6

Klaus Lehmann

@xomx great! glad, You have found an error. a misconfiguration bug.
I will wait for v8.6.6
than I can report some issues with speed. I want to prepare a comparison between n++ v8.6.6 and Vedit 6.2(64bit). more than later. 1st I will wait vor v8.6.6
Thanks for Your good job!
yours, klaus

Klaus Lehmann

@Klaus-Lehmann
my problem is solved. thank You for all!

in Notepad++ v8.6.6 Release Candidate:
http://download.notepad-plus-plus.org/repository/8.x/8.6.6.RC/
Change log:
Fix crash when crossing the 2GB file size threshold. (Fix #14944 , #14981 )

it works. great!
who is intested in: how looks a big file, opened with notpad++ v8.6.6(RC1)
notepad-8.6.6-RC1.jpg

its a interesting file ;-)
2,9 GB. including text in 67 Mill lines! mainly ascii

ok. some certain stuff:
my config.xml has this content
<GUIConfig name=“largeFileRestriction” fileSizeMB=“200” isEnabled=“no” allowAutoCompletion=“no” allowBraceMatch=“no” allowSmartHilite=“no” allowClickableLink=“no” deactivateWordWrap=“yes” suppress2GBWarning=“yes” />
and with these settings I have my colors back!
colors? I have changed "userDefineLang.xml. that I can see some special content, with “alarming colors”. You will it see in the picture. for me are the content very very certain, it begins with $0 or $a or or or.

ok. this all is very good for me! Thank You so much! ;-)
in next days I will do some tests. and publish them here!
notepad++ have to load a unknown file with approx. 3GB data. or with 11GB.
those big files are very welcome at me. I must do my work with them.
and without good tools it is VERY uncomfortable.
last 5-10 years I do my work with notepad++.
and for very very big files I do use -just now- VEDIT v6.2 as a 64bit. but notepadd++ is more comfortablier! ;-)

I remember: 5-10 years before, I only have used (former BRIEF) CRiSP. (and I think its known well from old dos-times.)
but just now? it seems CRiSP exists. but I don’t want take business with them, I have had “monetary” problems with the folks. Does anybody know more about CRiSP ???

Yours very, Klaus Lehmann

mkupper

@Klaus-Lehmann said in Big files cannot opened in v8.6.4 (x84):

VEDIT v6.2 as a 64bit. but notepadd++ is more comfortablier!

Why don’t you use the 64-bit version of Notepad++? It’s the same as the 32-bit version you are using. The only downside may be if you need a specific plugin that is only available as a 32-it version then, yes, you would need to use the 32-bit Notepad++.