How to create textblocks in a LOG file

Andre Bergmann

Hello,
I’ve LOG files which always contain large textblocks with these markers:
06:48:01,550 ERROR …
…
… XX more
[LF]

Each of these entries contains dozend of lines because of the whole LOG is confusing.

Maybe somebody has an idea if it will be possible to search for all of these blocks, make them to textblocks and collapse the created textblocks?

I’m not very familar with Notepad++ yet so I hope somebody can help with.
Thank you in advance!
Andre

Alan Kilborn

@andre-bergmann said in How to create textblocks in a LOG file:

Maybe somebody has an idea if it will be possible to search for all of these blocks, make them to textblocks and collapse the created textblocks?

I have no solid idea of what this means.
If you can explain further then maybe some help could be provided.

PeterJones

@andre-bergmann ,

Like @Alan-Kilborn , I am not really sure what you want. But it sounds like you want some sort of code folding for a specific LOG file format.

Using the User Defined Language (UDL) system, you can define code folding, with some limitation: There must be a distinguishable START and END text sequence to define the START and END of the block – like #if and #endif or { and } – having a folding block end with just a blank line (which I think is what you want) will not work in the UDL implementation; and a change in indentation (like Python uses for defining blocks) will not work in the UDL system.

If UDL’s code-folding is not sufficient for you, you may have to write (or hire/bribe someone to write) a full lexer plugin for your LOG language – a compiled lexer is able to establish any kind of folding you want, at the cost of you have to be able to write a whole lexer, rather than just typing in keywords or symbols that mark the beginning and end of blocks.

UDL Overview: https://npp-user-manual.org/docs/user-defined-language-system/
UDL details: https://ivan-radic.github.io/udl-documentation/
- including a page on cold folding: https://ivan-radic.github.io/udl-documentation/folding-in-code/

Andre Bergmann

Thank you for replies.

Using the User Defined Language (UDL) system, you can define code folding, with some limitation: There must be a distinguishable START and END text sequence to define the START and END of the block – like #if and #endif or { and } – having a folding block end with just a blank line (which I think is what you want) will not work in the UDL implementation; and a change in indentation (like Python uses for defining blocks) will not work in the UDL system.

Yes this is what I mean.
Thank you in general it works fine if the block definition START “ERROR” an END “more”.
Only in some cases the END tag “more” is more than one times included in the textblock.
The goal would be to define END of block with “more” and also including two CRLF because in this case I’ll be sure to have the right “more”.
I did some search but ((EOL)) did not work (I get this hint from here: https://community.notepad-plus-plus.org/topic/16963/newline-as-delimiter-in-a-udl/8).

Is the syntax “more((EOL))((EOL))” wrong?

PeterJones

@andre-bergmann said in How to create textblocks in a LOG file:

Is the syntax “more((EOL))((EOL))” wrong?

Inasmuch as it won’t work, it is wrong.

The ((EOL)) notation in UDL is meant to stand on its own, not be combined with something else. It’s purpose is for ending one of the “pairs” in UDL at the end of the current line – which would be meaningless for a block, which requires more than one line to work.

Fortunately for you, trying to come up with a workaround stuck with me, and I played with it in PythonScript a bit over the weekend. I got something that works tolerably well: you load your logfile, run this script to get it to make the block folding, then run it again to undo the block folding. With the new information of wanting more((EOL))((EOL)) to indicate the end of the block, I was able to finalize a regex that caught the end of block in my examples.

If you have the text

06:48:01,550 ERROR …
…
… XX more

something else

there were some blanks
06:55:01,550 ERROR …
… more here
… still more
… XX more

not part of the block

then when this script turns on the blocks, you will get:

And those folding blocks will work just like normal ones in Notepad++… except that if you edit the file, the blocks will no longer be valid. So the workflow with this implementation needs to be “load the LOG file; run the script to turn on blocks; edit; double-run the script to get updates to the folding regions”… which might be tolerable, but a little annoying. (the code could be changed to just always clear-then-redo the folding, rather than do the folding on the odd runs and clear the folding on the even runs)

It would be possible to use “notifications” in PythonScript to get it to run the code-folding function again after you edit, but depending on how often you make it run, Notepad++ might spend too many CPU cycles on running the PythonScript and not leave much CPU for your actual editing. If you did want to go the “notification” route, I would suggest notifying on something like a “save” event, rather than on every edit, which might be a compromise between making the blocks update regularly, but not so often that you cannot get anything done.

But the code as currently implemented is below. It includes instructions for how to install the plugin and this script in the comments.

# encoding=utf-8
"""in response to https://community.notepad-plus-plus.org/topic/22410/

This will apply simple (non-nested) folding to the active file.

It will apply a start_regex and end_regex, which define the start and
end of your "block" of LOGFILE that is meant to be folded.
The first time you run, it will set up all the folding based on those
regex; you can then use Notepad++'s fold/unfold mechanism to fold/unfold
those blocks.  The next time you run, it will clear the folding.  Repeat
as necessary.

Right now, this is _on demand_, not _on edit_, so if you edit your logfile,
the

Please note that if your file type already has an associated Lexer (programming Language)
or UDL (user defined language), these folds will conflict with the Lexer/UDL folding,
with unpredictable results.  Mix with caution.

___INSTRUCTIONS____

INSTALL:
0. If PythonScript is not yet installed,
    * Plugins > Plugins Admin
    * Check the ☑ PythonScript checkbox
    * Click INSTALL
1. Create an empty script:
    * Plugins > Python Script > New Script
    * Make sure the folder selected is
        `...notepad++\plugins\Config\PythonScript\scripts`
    * give it a name like LogFolding.py
2. Paste this whole script (including these instructions)
    into the file, and save
3. Edit the start_regex and end_regex right here:
"""
start_regex = r'^\d+:\d+:\d+,\d+ ERROR'
end_regex = r'more\R{2,}'
"""
    These need to be boost-compatible regular expressions

RUN:
1. Plugins > PythonScript > Scripts > LogFolding.py
or use the keyboard shortcut defined below

KEYBOARD SHORTCUT:
1. Plugins > PythonScript > Configuration
2. In the User Scripts, click on LogFolding.py
3. Click the left ADD button (above Menu Items)
    * This adds "LogFolding" to the main Plugins > PythonScript menu, rather
      than requiring you to dig down into the Scripts submenu
4. Exit Notepad++ and restart
    * This is required for ShortcutMapper to be able to see the new entry
      in the PythonScript menu
5. Settings > ShortcutMapper > Plugin Commands
6. Filter by LogFolding
7. Click on LogFolding, then click the MODIFY button and
    set the keyboard shortcut in the dialog
"""
from Npp import *
from time import sleep

class topic22410_LogFolder(object):
    OUTSIDE = "OUTSIDE"
    INSIDE = "INSIDE"
    HEADER = "HEADER"
    _DEBUG_ = False

    def __init__(self, my_file):
        self.matching = topic22410_LogFolder.OUTSIDE
        self.folding_defined = False
        self.file = my_file
        self.editor = editor    # store the active editor
        if notepad.getLangType() == LANGTYPE.TXT:
            notepad.setLangType(LANGTYPE.USER)
            sleep(0.1)
        if topic22410_LogFolder._DEBUG_: console.write("Create folder[{}]\n".format(self.file))
        self.toggle_folds()

    def toggle_folds(self):
        self.folding_defined = not self.folding_defined
        if topic22410_LogFolder._DEBUG_: console.write(".toggle_folds() to {}\n".format(self.folding_defined))
        self.clear_folds()
        if self.folding_defined: self.make_folds()
        if topic22410_LogFolder._DEBUG_: console.write(".toggle_folds() done\n")

    def clear_folds(self):
        if topic22410_LogFolder._DEBUG_: console.write(".clear_folds() running\n")
        self.editor.forEachLine(lambda c,l,t: self.editor.setFoldLevel(l,0x400))
        if topic22410_LogFolder._DEBUG_: console.write(".clear_folds() done\n")

    def make_folds(self):
        #self.editor.setFoldLevel(0, 0x2400)
        #self.editor.setFoldLevel(1, 0x0401)
        if topic22410_LogFolder._DEBUG_: console.write(".make_folds() running\n")
        self.editor.forEachLine(self._per_line)
        if topic22410_LogFolder._DEBUG_: console.write(".make_folds() done\n")

    def _per_line(self, content, line_index, total_lines):
        p1 = self.editor.positionFromLine(line_index)
        p2 = self.editor.getLineEndPosition(line_index)
        pe = self.editor.getLength()
        #console.write("checking __{}__: ({}..{}) =~ {}\n".format(1+line_index, p1, p2, start_regex))
        self.editor.research(
            start_regex,
            lambda m: self._line_matches(m, line_index, topic22410_LogFolder.HEADER),
            0,      # flags
            p1,p2,  # start/end position
            1       # match just one instance
        )
        if self.matching == topic22410_LogFolder.HEADER:
            # since this line is header,
            # set the folding to header-state (0x2400)
            self.editor.setFoldLevel(line_index, 0x2400)
            #console.write("\tHEADER @ {}\n".format(line_index))

            # but as soon as we're done with the header,
            # the state is now INSIDE
            self.matching = topic22410_LogFolder.INSIDE
        elif self.matching == topic22410_LogFolder.INSIDE:
            # since this line is INSIDE, need to set folding to 0x0401
            self.editor.setFoldLevel(line_index, 0x0401)
            #console.write("\tINSIDE @ {}\n".format(line_index))
        elif self.matching == topic22410_LogFolder.OUTSIDE:
            # once we are outside, we don't need to change the
            # state of the active line, because it was already reset
            #console.write("\tOUTSIDE @ {}\n".format(line_index))
            pass

        self.editor.research(
            end_regex,
            lambda m: self._line_matches(m, line_index, topic22410_LogFolder.OUTSIDE),
            0,      # flags
            p1,pe,  # start/end position: for end_regex, match to end-of-file rather than just the single line
            1       # match just one instance
        )

        #console.write("\t{:<7s}\tlevel={:#06x}\n".format('__{}__'.format(line_index),self.editor.getFoldLevel(line_index)))


    def _line_matches(self, m, line_index, match_state):
        pos_eol = self.editor.getLineEndPosition(line_index)
        #console.write("MATCH {}:\t__{}__:({}..{}): {}[EOL={} {}]\n".format(match_state,line_index,m.start(0),m.end(0),m.group(0).strip(),pos_eol,m.start(0)<=pos_eol))
        if m.start(0) <= pos_eol: # end of match must be before EOL to change state
            self.matching = match_state


if __name__ == "__main__":
    my_file = notepad.getCurrentFilename()
    try:                                # see if dictionary and key exist, and try to toggle
        folderObjects[my_file].toggle_folds()
    except NameError:                   # if dictionary not yet initialized
        #console.clear()
        folderObjects = {}              #   create the dictionary
                                        #   and populate key
        folderObjects[my_file] = topic22410_LogFolder(my_file)
    except KeyError:                    # if dictionary key does not yet exist
                                        #   then populate key
        folderObjects[my_file] = topic22410_LogFolder(my_file)

Andre Bergmann

@peterjones amazing thank you so much.
I’ll give it a try tomorrow but based on your description you ge my vision what I need.

So I can save a couple of hours I’ve not to reard hugh LOG files to find the main tags.

Alan Kilborn

@andre-bergmann said in How to create textblocks in a LOG file:

So I can save a couple of hours I’ve not to reard hugh LOG files to find the main tags.

This phrase makes me want to suggest something I’ve discovered to be useful recently: the Analyse plugin. It isn’t quite the same, but I think its original design was to make log file data easier to visually process. Just a suggestion.