Need help with custom sort operations

Dean-Corso

Yes, English isn’t my main language and I need to translate sometimes a little or just try to speak more in freestyle you know.

Ok, sounds bad that npp can’t do that at the moment or you don’t know any plugin what could handle it that way I’m looking for. So the principle of the sort operation I want is pretty simple (sort function npp has already I want to use / line operations) but only need that specific line-select / line-move thing etc. Maybe it would be a good idea to add another function in npp itself for that in any next release. Otherwise I also don’t have a clue how to do my sort operations I want in npp.

MfG

Terry R

@Dean-Corso said in Need help with custom sort operations:

So the principle of the sort operation I want is pretty simple

Well, sorry to say but what you need isn’t simple. If you had every line sorting alphabetically or numerically, ascending or descending then that is what I would call simple. And NPP covers those bases very well.

Beyond that there are so many different ideas (your’s being just 1 of thousands) I doubt the developer will consider expanding the NPP sorting functions to cover your’s or anyone else’s.

At present since you may have multiple block heading lines (in a single block) which need sorting first, before the new first block heading line becomes the line to sort the order of the blocks (your example 2) I can only think it will require some code (such as in PythonScript) to do what you need.

Unfortunately we aren’t a code creating forum as such, just a general forum supporting NPP users. Someone here might possibly consider it a worthwhile exercise to create such a program for you, but don’t expect it.

I will continue to ponder your request (I’m not a code creator) just in case there might be another way using just a (series of) regex.

Good luck
Terry

Mark Olson

This is a great opportunity to try out my new script for sorting multi-line blocks!

Just use the regex (?s-i)(^[A-Z]+$).*?(?=^[A-Z]+$|\Z) and the options

Group 1 transformation:  [   ]Integer(DEFAULT)      [   ]Decimal      [  x ]String      [   ]Ignorecase-cultural
Sort order:  [   ]Ascending(DEFAULT)    [   ]Descending
Other:
[   ]Sort-only-if-all-lines-have-group-1-match(DEFAULT)
[   ]Remove-non-matching-lines
[   ]Sort-non-matching-lines-to-bottom
[x   ]Multi-line-blocks

EXAMPLE:

DDDDDDDDDDDDD
onro
roeri

AAAAAAAAAAAAAAAAAAA
FFFFFFFFFFFFFFFFFFF
foo
EEEEEEEEEEEEEEEEEEEEE
oroieoj
roernern

reroo
BBBBBBBBBBBBBBBBBBB
ccc
furr
CCCCCCCCCCCCCCCCC

WILL BE SORTED TO

AAAAAAAAAAAAAAAAAAA
BBBBBBBBBBBBBBBBBBB
ccc
furr
CCCCCCCCCCCCCCCCC
DDDDDDDDDDDDD
onro
roeri
EEEEEEEEEEEEEEEEEEEEE
oroieoj
roernern

reroo
FFFFFFFFFFFFFFFFFFF
foo

Terry R

@Mark-Olson

I did just see your “other” post where you showed the new script and wondered if it would be any good for this.

I have a problem with your example as you show the first block as just having the DDDDD… line but when it’s sorted it suddenly gains the CCCCCC… line which was at the bottom in the before example. Was that a typo?

Or maybe I am just not reading the before and after examples right.

Terry

PS I was working on a poor mans method which was a bunch of regexes and using some of the more advanced caret moves to expand selections prior to sorting. The macros created would “run until end of file”.

Terry R

@Terry-R said in Need help with custom sort operations:

Or maybe I am just not reading the before and after examples right.

@Mark-Olson

I was in the headspace of working the solution based on 2 block header lines per block. When I saw your example it seemed like you were doing the same. Re-reading it, it suggests your example doesn’t have a defined block line length. I suppose that is good in that it can cater for a “loose” structure. But how does it know where each block starts and ends? (I’m not a code reader).

Terry

Mark Olson

@Terry-R said in Need help with custom sort operations:

Or maybe I am just not reading the before and after examples right.

Yeah, you’re just reading them wrong. My block-sorter just removed a trailing newline after the DDDDDDDDDDDD block.

Mark Olson

@Terry-R said in Need help with custom sort operations:

But how does it know where each block starts and ends? (I’m not a code reader).

That is a good question, which the documentation in my script should definitely address.

The answer is that your regex must match the entire block and have a single capture group within the portion to be matched.

Thus, (?s-i)(^[A-Z]+$).*?(?=^[A-Z]+$|\Z) is the appropriate regex to use, because the first capture group (^[A-Z]+$) contains the header line, and the regex as a whole matches everything up to the next header.

Terry R

@Mark-Olson said in Need help with custom sort operations:

My block-sorter just removed a trailing newline after the DDDDDDDDDDDD block.

If you look at the first post the example there showed a “trailing” blank/empty line. They wanted this kept.

From all @Dean-Corso posts so far I think they want the ability to dictate 2 parameters before running the process.

How many lines in each block. They state they are well defined, so ALL the same number of lines.
How many of the lines (from start of block down) will be sorted first within the block.

Lastly another sort is performed on the file using ONLY the first header line of each block (keeping each block intact).

Currently I don’t think your script will work that correctly, certainly as you seem to be removing “blank” lines that won’t work for them.

Terry

Mark Olson

@Terry-R said in Need help with custom sort operations:

If you look at the first post the example there showed a “trailing” blank/empty line. They wanted this kept.

Yep, that’s a limitation of my script. I’m not going to change it, because every way I’ve tried to get around that limitation makes things worse.

Fortunately, it is quite easy to make a regex-replace that creates a trailing blank/empty line after each block. Just replace (?<!\A)(^[A-Z]+$) with \r\n{0}

Mark Olson

@Dean-Corso said in Need help with custom sort operations:

Just need to have some function / dialog where I can enter those paramters…
Enter which every line to sort: 2
Enter how many lines to sort: 1

Done. Just use the following script in PythonScript

# -*- coding: utf-8 -*-
#########################################
#
#  sort_line_blocks_and_headers
#
#########################################

# references:
#  https://community.notepad-plus-plus.org/topic/24742/need-help-with-custom-sort-operations

# HOW IT WORKS:
# 1. Divide the document into blocks of N lines, of which the top M lines are the header.
# 2. Sort the header of each block
# 3. Sort the blocks by the sorted headers.
# EXAMPLE INPUT:
# CCCCCCCCCCCCCCCCCCC
# AAAAAAAAAAAAAAAAAAA
# 3
# BBBBBBBBBBBBBBBBBBB
# AAAAAAAAAAAAAAAAAAA
# 2
# CCCCCCCCCCCCCCCCCCC
# BBBBBBBBBBBBBBBBBBB
# 1
# RESPOND TO THE 4 PROMPTS AS FOLLOWS: 3, yes, 2, yes
# RESULT:
# AAAAAAAAAAAAAAAAAAA
# BBBBBBBBBBBBBBBBBBB
# 2
# AAAAAAAAAAAAAAAAAAA
# CCCCCCCCCCCCCCCCCCC
# 3
# BBBBBBBBBBBBBBBBBBB
# CCCCCCCCCCCCCCCCCCC
# 1
from __future__ import print_function
from Npp import *

def main():
    eol = [ '\r\n', '\r', '\n' ][ editor.getEOLMode() ]
    while True:
        lines_per_block_str = notepad.prompt('number of lines per block', 'sort blocks', '3')
        if lines_per_block_str is None:
            return
        try:
            lines_per_block = int(lines_per_block_str)
            if lines_per_block < 1:
                raise ValueError
        except:
            notepad.messageBox('lines per block must be integer >= 1')
            continue
        sort_blocks_ascending_str = notepad.prompt('sort blocks least to greatest (yes/no)', 'sort blocks', 'yes')
        if sort_blocks_ascending_str is None:
            return
        reverse_blocks = sort_blocks_ascending_str != 'yes'
        lines_in_header_str = notepad.prompt('number of lines in header', 'sort blocks', '1')
        if lines_in_header_str is None:
            return
        sort_header_ascending_str = notepad.prompt('sort header least to greatest (yes/no)', 'sort blocks', 'yes')
        if sort_header_ascending_str is None:
            return
        reverse_header = sort_header_ascending_str != 'yes'
        try:
            lines_in_header = int(lines_in_header_str)
            if lines_in_header < 1 or lines_in_header > lines_per_block:
                raise ValueError
        except:
            notepad.messageBox('lines in header must be integer <= lines in block and >= 1')
            continue
        break
    lines_not_in_header = lines_per_block - lines_in_header
    find_regex = '(?-s)((?:.*\R){%s})' % lines_in_header
    if lines_not_in_header:
        find_regex += '((?:.*(?:\R|\Z)){%s})' % lines_not_in_header
    # print(find_regex)
    header_block_list = []
    doc_len = editor.getLength()
    
    def on_match(m):
        sorted_header = eol.join(sorted(m.group(1).splitlines(), reverse=reverse_header))
        edited_block = sorted_header + eol + m.group(2)
        if m.span()[1] < doc_len:
            edited_block = edited_block[:-len(eol)]
        header_block_list.append((sorted_header, edited_block))
    
    editor.research(find_regex, on_match)
    header_block_list_sorted = [x[1] for x in sorted(header_block_list, reverse=reverse_blocks)]
    # print(header_block_list)
    editor.setText(eol.join(header_block_list_sorted))
    
if __name__ == '__main__':
    main()

Mark Olson

@Mark-Olson
My initial script had some bugs. Use this instead

# -*- coding: utf-8 -*-
#########################################
#
#  sort_line_blocks_and_headers
#
#########################################

# references:
#  https://community.notepad-plus-plus.org/topic/24742/need-help-with-custom-sort-operations

# HOW IT WORKS:
# 1. Divide the document into blocks of N lines, of which the top M lines are the header.
# 2. Sort the header of each block
# 3. Sort the blocks by the sorted headers.
# EXAMPLE INPUT:
# CCCCCCCCCCCCCCCCCCC
# AAAAAAAAAAAAAAAAAAA
# 3
# BBBBBBBBBBBBBBBBBBB
# AAAAAAAAAAAAAAAAAAA
# 2
# CCCCCCCCCCCCCCCCCCC
# BBBBBBBBBBBBBBBBBBB
# 1
# RESPOND TO THE 4 PROMPTS AS FOLLOWS: 3, yes, 2, yes
# RESULT:
# AAAAAAAAAAAAAAAAAAA
# BBBBBBBBBBBBBBBBBBB
# 2
# AAAAAAAAAAAAAAAAAAA
# CCCCCCCCCCCCCCCCCCC
# 3
# BBBBBBBBBBBBBBBBBBB
# CCCCCCCCCCCCCCCCCCC
# 1
from __future__ import print_function
from Npp import *

def main():
    eol = [ '\r\n', '\r', '\n' ][ editor.getEOLMode() ]
    
    while True:
        lines_per_block_str = notepad.prompt('number of lines per block', 'sort blocks', '3')
        if lines_per_block_str is None:
            return
        try:
            lines_per_block = int(lines_per_block_str)
            if lines_per_block < 1:
                raise ValueError
        except:
            notepad.messageBox('lines per block must be integer >= 1')
            continue
        
        sort_blocks_ascending_str = notepad.prompt('sort blocks least to greatest (yes/no)', 'sort blocks', 'yes')
        if sort_blocks_ascending_str is None:
            return
        reverse_blocks = sort_blocks_ascending_str != 'yes'
        
        lines_in_header_str = notepad.prompt('number of lines in header', 'sort blocks', '1')
        if lines_in_header_str is None:
            return
        try:
            lines_in_header = int(lines_in_header_str)
            if lines_in_header < 1 or lines_in_header > lines_per_block:
                raise ValueError
        except:
            notepad.messageBox('lines in header must be integer <= lines in block and >= 1')
            continue
        
        if lines_in_header > 1:
            sort_header_ascending_str = notepad.prompt('sort header least to greatest (yes/no)', 'sort blocks', 'yes')
            if sort_header_ascending_str is None:
                return
            reverse_header = sort_header_ascending_str != 'yes'
        else:
            reverse_header = False
        break
    
    lines_not_in_header = lines_per_block - lines_in_header
    if lines_not_in_header:
        find_regex = '(?-s)((?:.*\R){%s})((?:.*(?:\R|\Z)){%s})' % (lines_in_header, lines_not_in_header)
    else:
        find_regex = '(?-s)((?:.*(?:\R|\Z)){%s})' % lines_in_header
    # print(find_regex)
    header_block_list = []
    doc_len = editor.getLength()
    
    def on_match(m):
        is_last_match = m.span()[1] == doc_len
        group1 = m.group(1)
        if not group1:
            return
        sorted_header = eol.join(sorted(group1.splitlines(), reverse=reverse_header))
        if lines_not_in_header:
            edited_block = sorted_header + eol + m.group(2)
        else:
            edited_block = sorted_header
        if not is_last_match and edited_block.endswith(eol):
            edited_block = edited_block[:-len(eol)]
        header_block_list.append((sorted_header, edited_block))
    
    editor.research(find_regex, on_match)
    header_block_list_sorted = [x[1] for x in sorted(header_block_list, reverse=reverse_blocks)]
    # print(header_block_list)
    editor.setText(eol.join(header_block_list_sorted))
    
if __name__ == '__main__':
    main()

Dean-Corso

@Terry-R
@Mark-Olson

Thank you guys for trying to help me (and others maybe too) to find a solution for my sorting problems. Sounds great that you was able to create a script for that task. Only problem at the moment is I don’t know yet how to use it so maybe you could give me a little crash curse what plugin to install and what parameters I have to execute (python + your script + my npp tab or file) etc. So I have python installed already on computer. My question is also whether I can execute your py script in npp for the open / focused tab itself (also when I just have written text in real time without saved file). As I said, not sure what plugin to install for py (by the way, the PyNPP plugin is not listed in plugin manager) if I have to. Would be nice if you could tell me and show one or two examples how to execute your script with an file which is open in notepad++ or if possible also using a direct tab (none file) in pipe mode etc if this is supported. Thank you.

PeterJones

@Dean-Corso said in Need help with custom sort operations:

little crash curse what plugin to install and what parameters I have to execute

FAQ: How to install and run a script in pythonscript

Dean-Corso

@PeterJones

Thank you for that link & info. Ok, I think I got it working now to execute your script @Mark-Olson and on my first tests it seems to work as I wanted! Coolio! I will do some more tests but it looks already pretty well.

Is this sort operation which works now for all complete lines also doable to for that rectangle selection (maybe for later)? Lets say I have few vertical text blocks and I want to select any of them via rectangle selection and want to sort all lines from that selection. Could that be possible too?

Just as info: I found out that if I want to sort specific rectangle selected lines (middle - end) that it’s only working when I trim all the selected lines starting at same vertical position and I also need to change all blank chars like TABs to Space before I do a sort to make it work. Just don’t understand why but I have to.

Example: 2 blocks / tabs between / sort ascending ic.

66666				qqqqqqq
55555555			zzzzzzz
777					aaaaaaa
88888888888			xxxxxxx

I just select the second block via rectangle select then I do the line sort with ingore case and I got this out.

55555555			zzzzzzz
88888888888			xxxxxxx
777					aaaaaaa
66666				qqqqqqq

=? So when I do same but before I call the blank option “Tab to Space” and select the 2. block and call then the sort function then I get this out…

777                 aaaaaaa
66666               qqqqqqq
88888888888         xxxxxxx
55555555            zzzzzzz

which is correctly now. Just don’t understand why I have to change the tabs to space before the selection so normally it should play no role at this point what comes before the selection you know. Is this a bug or do I understand it wrong? You know what I mean?

Thank you very much so far guys.

Coises

@Dean-Corso If it helps you, my plugin Columns++ (still in alpha stage, though apparently stable) can sort column selections containing tabs.

The reason the native sorts in Notepad++ don’t work is because they determine the first and last character positions of the rectangular selection in the first line of the selection and then apply those character positions to all lines; they don’t use the line-by-line contents of the rectangular selection.

Dean-Corso

@Coises

Thank you for that info. I did test your plugin and it seems to work to do the sort operation without conversion. I also see your plugin has also some more interesting functions I need to check out more.

I think one of the problem in npp & plugin is that in both are missing some clear methods to use specific operations just for all lines OR selection with or without to handle lines you know what I mean?

Example: Sort function.
1.) I want to handle all lines
2.) I want to handle just my selection (normal case)
3.) I want to handle rectangle selection + lines
4.) I want to handle rectangle selection itself only

Point 1 & 2 seems to work good so far in npp to do that but in special cases 3 & 4 it could be problematic.

When I sort any lines (selected or not) then empty lines getting removed. Why? How to prevent that? On first view it looks good to remove blank lines but in case when I want to keep the lines original then I need to add empty lines manually after sorting.

What is when I just want to sort the content of my rectangle selection without to move all the lines? I see its also not doable so far in npp.

What is when I want to sort my rectangle selected part / lines and ignore all blank chars till the first char begins? Not working as I can see and I need to bring all the selected lines starting at the same position point like this…(I did rectangle select the right part)

66666		Qqqqqqq
55555555				Zzzzzzz
777				Aaaaaaa
88888888888		Xxxxxxx

…and before I can sort I need to change them to this…

66666               Qqqqqqq
55555555            Zzzzzzz
777                 Aaaaaaa
88888888888         Xxxxxxx

…and now I can do a sort which works and moves all lines. But what if I want just to sort that selection only and keep the left part / lines unchanged? Like this…

66666               Aaaaaaa
55555555            Qqqqqqq
777                 Xxxxxxx
88888888888         Zzzzzzz

…only selection was sorted and the rest is same. So is it too specific what I want?

Coises

@Dean-Corso said in Need help with custom sort operations:

Example: Sort function.
1.) I want to handle all lines
2.) I want to handle just my selection (normal case)
3.) I want to handle rectangle selection + lines
4.) I want to handle rectangle selection itself only

Point 1 & 2 seems to work good so far in npp to do that but in special cases 3 & 4 it could be problematic.

Case 3 should work using Columns++. It also works with the built-in Notepad++ sorts provided you have a fixed-pitch font and no tabs.

Case 4 can be handled easily this way:

Make the rectangular selection.
Copy the selection to the clipboard.
Open a new tab and paste the selection into it.
Sort the new tab.
Make a rectangular selection encompassing all the data.
Copy the selection to the clipboard.
Switch back to the original tab and paste the data into the rectangular selection.

Doing this in one step, instead of copying and pasting into a temporary tab, is something I could include in Columns++. I can’t promise, but the next time I make changes I’ll see if it can be added without making things too clumsy or complicated.

When I sort any lines (selected or not) then empty lines getting removed. Why? How to prevent that? On first view it looks good to remove blank lines but in case when I want to keep the lines original then I need to add empty lines manually after sorting.

What is when I just want to sort the content of my rectangle selection without to move all the lines? I see its also not doable so far in npp.

That’s a bit more complicated. The blank lines aren’t being removed, they’re just sorting, the same as all the other lines; to the beginning if it’s an ascending sort or to the end if it’s a descending sort. If you want to skip sorting those lines, then it becomes confusing. Do they serve as breaks — meaning each section between blank lines is sorted, but lines aren’t moved across blank lines? Or do all the non-blank lines get sorted, but then positioned only where non-blank lines occurred before, leaving the blank lines at the same line number positions? Or do the blank lines “stick with” the preceding non-blank line? Or the following non-blank line? Or something else?

I think there are too many variations in what someone could want to make it practical to implement.

What is when I want to sort my rectangle selected part / lines and ignore all blank chars till the first char begins? Not working as I can see and I need to bring all the selected lines starting at the same position point like this…(I did rectangle select the right part)
66666		Qqqqqqq
55555555				Zzzzzzz
777				Aaaaaaa
88888888888		Xxxxxxx
…and before I can sort I need to change them to this…
66666               Qqqqqqq
55555555            Zzzzzzz
777                 Aaaaaaa
88888888888         Xxxxxxx

At present, none of the sorts have an option to ignore leading blanks, but that is a logical request. For this, too, I’ll say that when I next make some changes to Columns++, I will see if something like this can work. Offhand, I think it might be handy to make a sort that applies a user-specified regular expression to each line in a rectangular selection to determine the sort key. Skipping leading blanks would be an easy special case of that.

Dean-Corso

@Coises

Thank you for the feedback. Yes I have seen that I can copy / paste the rectangle selection using another tab or free space below etc but this is again a detour I want to prevent of course. Sounds good if you could add some more features in your plugin if possible. I already added your plugin releases into my RSS feed list.

I was trying around to handle specific tasks I got in my mind when editing different text and lines using Notepad++ and have seen that not all I wanna do is doable yet. Would be good if you could add more features into your plugin later to have more capabilities & possibilities to edit text, lines and selections etc. Just another suggestion to you @Coises, could you also add that specific sort method (block sort / move) I did requested and @Mark-Olson made that python script for that. Maybe you can adapt it and include it into your plugin sort operation with extra custom dialog to enter the user data. Thanks.

PS: Just a question about this forum style. How to reply & see the forum at the same time? Each time when I wanna reply an answer then the text field does popup as overlay and I can’t see the text of the posts anymore and I can just move this reply window a little bit down only. I would like to see all at once like in other forums too. How to do that here?

Mark Olson

@Dean-Corso
These are very interesting ideas! Many of them are pretty tricky to implement. The good news is that if you are feeling adventurous and want to learn some useful skills, you can begin trying to implement such solutions yourself by learning regular expressions and Python. By the way, this is not me telling you to leave us alone, but rather a general observation that sometimes if you want something done right you just have to do it yourself.
I can’t speak for Coises, of course, but a few suggestions about feature requests:

There are many excellent features that are simply beyond what the developer envisions as the scope of their application. For instance, I have a JSON plugin. A user might say, “hey, I’d like a CSV parser so I can use your tree view for CSV files as well!” And that would be really cool! The trouble is that a CSV parser is just outside the scope of my plugin, and it opens the door for even more “feature creep”.
I have often created a script solution that seemed to me like it solved two or three problems simultaneously, and I would never have to do this complicated multi-step process again. Instead, I created a fragile tool that solved two or three problems badly, and I often wound up having to do the multi-step process anyway because I could use more effective specialized tools for each step.

Dean-Corso

@Mark-Olson

So I know what you mean and learning some helpfully language +/- Regular Expression would make me more independent to handle specific tasks by myself in the best case. I tried to keep RegEx Syntax in my mind but always forget it when not using it anymore for a longer while. Also have problem with those different syntax styles I have to use here or there what makes it also more problematic for me. Will see what I can do with python script in the future. Otherwise I will ask you guys again to get some help if you don’t mind.

Your script works very good so far and didn’t found any issues yet. I also made a quick accessible icon in the toolbar to prevent the long menu choosing method. Thank you.