7.7.1 breaks Edit > Line Operations > Sort Lines As Integers Ascending for CIDR IPv4

PeterJones · Jun 26, 2019, 9:16 PM

Unsorted para confuses me. It’s not what I do or did or even want to do.

So take a step back here: I believe there are really two activities going on simultaneously, which you need to separate in your mind. 1) You have a complicated, huge set of data which you want to sort. 2) You’ve found a bug in the sorting algorithm, which the 13 lines of data evidence.

For #1: Until the bug is fixed, or unless you revert to v7.7, you are not going to get your actual huge data sorted. If you need that done quickly, go download v7.7 and run using that (if you don’t want to uninstall v7.7.1, then you can download the portable “zip” version, and export to a folder (on your desktop or similar) and run from there to get your task done. Thus, ignore #1 for now

So, on to #2: accurately and helpfully reporting the bug. To do that in an unconfusing manner, you need to supply a small set of unsorted data which will give the different results depending on whether it’s in v7.7 or v7.7.1. The thirteen lines I showed in “unsorted” is just such a set. Of course you don’t want this data in this order: this is just a starting point that will show the bug. At this point, remember, all we’re considering is the bug itself, not whether the data matches your full data set. Of course it’s not properly sorted: you need to start with unsorted data to show a bug in a sorting algorithm. Of course it doesn’t have all the extra data in your original data: it’s a short, self-contained data set that shows all the issues with little extraneous information.

Sorted as integers para is unavailable to me as I don’t have 7.7 any more.

And I’ve kindly given you the result of sorting in v7.7. So you don’t need v7.7… though you could grab as easily as I grabbed v7.7.1 portable/zip to be able to confirm the v7.7.1 sort-order after my last post.

Sorted as inters [sic] para is another way of putting my original premis.

Sorry for the typo; I meant “Sorted as inters in v7.7.1 (unwanted behavior)”, of course. And that paragraph was meant to match your original problem statement. Note that in your github issue, you didn’t even give that much information, which was one of my points to you.

Its last line is incorrectly ordered.

Yep, the unsorted data shows the bug in more than one location. I just highlighted the first one I noticed.

Also I pointed out exactly where it’s going wrong too.

Except you had no data; you didn’t show what you started with, what you expected, or what you were wrongly getting. You gave some words with jargon/abbreviations. Not all programmers deal with “CIDR” or “raw ASN listing”, and so you may be asking them to go look up what those terms are, which seems less helpful than supplying the 13 lines of data that you originally posted here (or the three sets of 13 lines which truly show the whole issue, unsorted, wrongly sorted, and desired sort order); not all software development teams handling bug reports want to come up with their own data set which may or may not evidence the bug being reported, when it would be so much simpler if the person reporting the bug also included data.

I need to go away and fix …

Good luck.

piranpiran · Jun 27, 2019, 1:51 AM

@chcg said:

Maybe https://github.com/notepad-plus-plus/notepad-plus-plus/commit/ff20c264df4167943fff6247fec4b0c0ce6227fb#diff-4608be755b00f4ec444233203ee8eafc changed the behaviour as it is a change to the sorting introduced from 7.7 -> 7.7.1.

Good point.

[18446744073709551615] :: http://www.cplusplus.com/reference/climits/
“ULLONG_MAX - Maximum value for an object of type unsigned long long int - 18446744073709551615 ((2 to the 64) -1) or greater”
OK, a big number …issue may depend on how IPv4 octet max 255.255.255.255 is now being interpreted or parsed?

[7.7.1 change notes] :: “Fix crash while sorting lines with numbers longer than 20 digits” :: I’ve used the ‘previous’ sort mechanism in npp intensively daily for months. No crashes. Sort is reliable & quick. System copes. Something else in 7.7.1 changed or is flawed?

Alan Kilborn · Jun 27, 2019, 1:51 AM

@piranpiran said:

Fix crash while sorting lines with numbers longer than 20 digits

Apparently someone noticed a problem with really long (as in digits) numbers and an attempt was made to fix that. A side effect was that it broke the undocumented behavior you were used to. It remains to be seen if the developers think your desired undocumented behavior is worth bringing back.

piranpiran · Jun 27, 2019, 1:58 AM

Too tired now. The workaround I suggested in the bug site thread did not satisfy at all times. I tried to edit it to no avail. Then I hit its preview delete button and now the whole bug report thread has been deleted or is unavailable (possibly another dragon). If a mod can restore the bug report then good otherwise I’ll have to try and retrace steps after some decent sleep. If anyone’s interested. Seems like a minority vertical issue and it’s just me reporting:-/ g’night

guy038 · Jun 27, 2019, 1:17 PM

Hello, @piranpiran, and All,

As I’ve verified that lexicographically sort is NOT broken in the v7.7.1 release, and acts exactly like the prior releases, you could use a regex S/R which would change the IPV4 addresses in such a way that a lexicographically sort would be possible !!

Let’s have a try on your small sample text, below, pasted in a new N++ tab :

13.228.0.0/15 REACH
15.177.48.0/21 Amazon ROUTE53
18.140.0.0/15 Amazon EC2 SIN prefix source:RADB
35.154.0.0/16 Amazon EC2 BOM prefix source:RADB
43.250.192.0/24 Amazon Asia-Pacific Resources
13.228.0.0/15 Amazon EC2 SIN prefix source:RADB
18.138.0.0/15 Amazon EC2 SIN prefix source:RADB
18.142.0.0/15 Amazon EC2 SIN prefix source:RADB
43.250.193.0/24 Amazon Asia-Pacific Resources
13.250.0.0/15 Amazon EC2 SIN prefix source:RADB
43.250.193.0/24 Amazon SIN Prefix source:RADB
18.136.0.0/16 Amazon EC2 SIN prefix source:RADB
43.250.192.0/24 Amazon SIN prefix source:RADB

Open the Replace dialog ( Ctrl + H )

SEARCH \b(((\d)?\d)?\d)(?=\.|/)
REPLACE (?2(?3:\x20):\x20\x20)\1
Tick, preferably, the Wrap around option
Select the Regular expression search mode
Click on the Replace All button

Et voilà ! You should get the following text :

 13.228.  0.  0/15 REACH
 15.177. 48.  0/21 Amazon ROUTE53
 18.140.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 35.154.  0.  0/16 Amazon EC2 BOM prefix source:RADB
 43.250.192.  0/24 Amazon Asia-Pacific Resources
 13.228.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 18.138.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 18.142.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 43.250.193.  0/24 Amazon Asia-Pacific Resources
 13.250.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 43.250.193.  0/24 Amazon SIN Prefix source:RADB
 18.136.  0.  0/16 Amazon EC2 SIN prefix source:RADB
 43.250.192.  0/24 Amazon SIN prefix source:RADB

This regex adds the appropriate number of space character(s) to any number, before the / symbol, containing from 1 to 3 digits, in order to align the four blocks of an IPV4 address ;-))

Now, after using the Edit > Line Operations > Sort lines Lexicographically Ascending menu option, you should be left with your expected sort :

 13.228.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 13.228.  0.  0/15 REACH
 13.250.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 15.177. 48.  0/21 Amazon ROUTE53
 18.136.  0.  0/16 Amazon EC2 SIN prefix source:RADB
 18.138.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 18.140.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 18.142.  0.  0/15 Amazon EC2 SIN prefix source:RADB
 35.154.  0.  0/16 Amazon EC2 BOM prefix source:RADB
 43.250.192.  0/24 Amazon Asia-Pacific Resources
 43.250.192.  0/24 Amazon SIN prefix source:RADB
 43.250.193.  0/24 Amazon Asia-Pacific Resources
 43.250.193.  0/24 Amazon SIN Prefix source:RADB

Finally, to get the initial layout of your IPV4 list, use the simple regex S/R, below :

SEARCH (?-s)\x20+(?=.*/)

REPLACE Leave EMPTY

Here we are !

13.228.0.0/15 Amazon EC2 SIN prefix source:RADB
13.228.0.0/15 REACH
13.250.0.0/15 Amazon EC2 SIN prefix source:RADB
15.177.48.0/21 Amazon ROUTE53
18.136.0.0/16 Amazon EC2 SIN prefix source:RADB
18.138.0.0/15 Amazon EC2 SIN prefix source:RADB
18.140.0.0/15 Amazon EC2 SIN prefix source:RADB
18.142.0.0/15 Amazon EC2 SIN prefix source:RADB
35.154.0.0/16 Amazon EC2 BOM prefix source:RADB
43.250.192.0/24 Amazon Asia-Pacific Resources
43.250.192.0/24 Amazon SIN prefix source:RADB
43.250.193.0/24 Amazon Asia-Pacific Resources
43.250.193.0/24 Amazon SIN Prefix source:RADB

This second regex searches for any non-null amount of space characters, located before the / symbol and delete them, as the replacement zone is Empty

I just hope that these two regex S/R work nice, too, on your VERY BIG unordered ASN data list ;-))

Best Regards,

guy038

AngryGamer · Jun 27, 2019, 11:24 AM

@piranpiran The dragon is slayed. Rest easy. PR#5843 😛

Alan Kilborn · Jun 27, 2019, 1:08 PM

I’d suggest that the OP test the process put forward by @guy038 and, if it validates, record these actions into a macro for future use. As long as the sort algorithm in N++ 7.7.1 sorts true integers correctly, I say there is no bug and the code should be left alone.

The dragon is slayed. Rest easy PR#5843

Not sure there was a dragon to slay, other than the posting dragon for those with reputation < 2. :)

AngryGamer · Jun 27, 2019, 1:09 PM

@Alan-Kilborn False. No matter what there’s a dragon to slay here. Re-check PR#5843 to see why the v7.7.1 setup is a complete mess. I’ve added more information to it.

piranpiran · Jun 27, 2019, 1:17 PM

@guy038 Very thoughtful, thank you for that work and I commend your positive attitude. I have never actually used the lexicographic option before now. The integer option, as it behaved prior to 7.7.1, ‘just worked’ - brilliantly effectively and blindingly quickly - with nary a crash despite the apparent citation in 7.7.1 change notes.

piranpiran · Jun 27, 2019, 1:33 PM

@guy038 said:

I just hope that these two regex S/R work nice, too, on your VERY BIG unordered ASN data list ;-))

Long time ago I used to throw a dozen or so S/R (?Scintilla?) keyboard macros at this situation. Quite buggy in their own right. Very slow and held up my workstation for minutes on end prompting Windows to pale out the screen and issue a soft-crash notification (which I learnt to ignore and just allow it all to finish properly IDC).

The ASN listing cited earlier is not particularly long compared with some of them out there which are truly monumentally sized. The S/R times became unreasonable to, frankly, largely risible. Complicated too.

Then I discovered the sort option behaviour cited by this thread… so quick, so simple, so WOW:-0

I am therefore loathe to go back to needing to use S/R hacks. If this issue cannot be resolved or worked around I very much think that I will need to adopt an earlier suggestion from @PeterJones to download the earlier ZIP and keep it on hand just to do the sorting. I had not known that this was feasible.

piranpiran · Jun 27, 2019, 1:45 PM

@guy038 You want a big unordered ASN listing? In my best Crocodile Dundee voice: THIS is a BIG one:
https://www.robtex.com/as/AS7552.html

guy038 · Jun 27, 2019, 6:33 PM

Hi, @piranpiran, and All,

As my laptop is quite an antiquity ( NEC Versa M-350, got, at work, in 2005, with XP SP3 and 1 GB of RAM only !! ), I don’t think that I’ve fully opened and pasted your AS7552.html, from my Firefox browser ;-))

But, never mind ! The part, that I’ve got, contains 291,800 lines, which begins and ends as below :

REGISTERED ROUTES
203.113.128.0/19 contact vieteladm@vietel.com.vn if necessary source:RADB
route	203.113.128.0/19
descr	contact vieteladm@vietel.com.vn if necessary
origin	AS7552
mnt-by	MAINT-AS3786
changed	yuncheol@bora.net 20021101
source	RADB
203.113.128.0/21 contact vieteladm@vietel.com.vn if necessary source:RADB
route	203.113.128.0/21
descr	contact vieteladm@vietel.com.vn if necessary
origin	AS7552
mnt-by	MAINT-AS3786
changed	yuncheol@bora.net 20021101
source	RADB
203.113.136.0/22 contact vieteladm@vietel.com.vn if necessary source:RADB
route	203.113.136.0/22
descr	contact vieteladm@vietel.com.vn if necessary
origin	AS7552
mnt-by	MAINT-AS3786
changed	yuncheol@bora.net 20021101
source	RADB
203.113.160.0/21 PACNET (proxy-registered route object) source:RADB
route	203.113.160.0/21
descr	PACNET (proxy-registered route object)
origin	AS7552
remarks	This route object is for a PACNET customer route which is
remarks	being exported under this origin AS.
remarks	+
remarks	This route object was created because no existing route
remarks	object with the same origin was found, and since some
remarks	PACNET peers filter based on these objects this route
remarks	may be rejected if this object is not created.
remarks	+
remarks	Please contact abuse@pacnet.net if you have any
remarks	Concerns regarding Spam/Abuses related to this object
remarks	+
remarks	Please contact ip-noc@pacnet.net if you have any other
remarks	Questions regarding this object.
notify	ip-noc@pacnet.net
mnt-by	MAINT-AS10026
changed	ip-noc@pacnet.net 20100201
source	RADB
........
........
........
2402:800:8320::/44 Prefix IPv6 Viettel-VN source:NTTCOM
route6	2402:800:8320::/44
descr	Prefix IPv6 Viettel-VN
origin	AS7552
notify	noc_ip@viettel.com.vn
mnt-by	MAINT-VIETTEL
changed	noc_ip@viettel 20190409
source	NTTCOM
2402:800:8330::/44 Prefix IPv6 Viettel-VN source:NTTCOM
route6	2402:800:8330::/44
descr	Prefix IPv6 Viettel-VN
origin	AS7552
notify	noc_ip@viettel.com.vn
mnt-by	MAINT-VIETTEL
changed	noc_ip@viettel 20190409
source	NTTCOM
2402:800:8340::/44 Prefix IPv6 Viettel-VN source:NTTCOM
route6	2402:800:8340::/44
descr	Prefix IPv6 Viettel-VN
origin	AS7552
notify	noc_ip@viettel.com.vn
mnt-by	MAINT-VIETTEL
changed	noc_ip@viettel 20190409
source	NTTCOM

Again, with this new regex S/R :

SEARCH ^(?!.*/\d+\x20.+).+\R

REPLACE Leave EMPTY

I could isolate, after 28s, the interesting part, below. I’ve just displayed the beginning and end, of the resulting file, which contains 36387 lines. !

203.113.128.0/19 contact vieteladm@vietel.com.vn if necessary source:RADB
203.113.128.0/21 contact vieteladm@vietel.com.vn if necessary source:RADB
203.113.136.0/22 contact vieteladm@vietel.com.vn if necessary source:RADB
203.113.160.0/21 PACNET (proxy-registered route object) source:RADB
........
........
........
2402:800:8320::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8330::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8340::/44 Prefix IPv6 Viettel-VN source:NTTCOM

Now, I’ve improved the regex, which inserts space characters, in order to take IPv6 addresses in account, too ;-))

SEARCH (?:(^)|\b)(((\d)?\d)?\d)(?=\.|/)

REPLACE (?1\x20)(?3(?4:\x20):\x20\x20)\2

I simply added a space character, before any IPv4 address, to be sure that after sort, all IPv4 addresses are listed before all IPv6 addresses. The modification of the 36387 IPv4 addresses was done in 17s

Then, I used the Edit > Line Operations > Sort lines Lexicographically Ascending menu option ( 1s )

And, after removing all the extra space chars, from IPv4 addresses, with the following regex S/R :

SEARCH (?-s)\x20+(?=.*/)

REPLACE Leave EMPTY

We get, 8s later, the beginning, the IPv4-IPv6 junction and the end of the list, as below :

27.64.0.0/12 Proxy-registered route object source:RADB
27.64.0.0/12 route object of AS7552 source:NTTCOM
27.64.0.0/13 Proxy-registered route object source:RADB
27.64.0.0/13 route object of AS7552 source:NTTCOM
27.64.0.0/14 Proxy-registered route object source:RADB
27.64.0.0/14 route object of AS7552 source:NTTCOM
27.64.0.0/15 Proxy-registered route object source:RADB
27.64.0.0/16 Proxy-registered route object source:RADB
27.64.0.0/16 route object of AS7552 source:NTTCOM
27.64.0.0/17 Proxy-registered route object source:RADB
27.64.0.0/18 Proxy-registered route object source:RADB
27.64.0.0/18 route object of AS7552 source:NTTCOM
27.64.0.0/19 Proxy-registered route object source:RADB
27.64.0.0/20 Proxy-registered route object source:RADB
27.64.0.0/20 route object of AS7552 source:NTTCOM
27.64.0.0/21 Proxy-registered route object source:RADB
27.64.0.0/22 Proxy-registered route object source:RADB
27.64.0.0/23 Proxy-registered route object source:RADB
27.64.0.0/24 Proxy-registered route object source:RADB
............
............
............
220.231.123.0/24 Proxy-registered route object source:RADB
220.231.124.0/22 Proxy-registered route object source:RADB
220.231.124.0/22 route object of AS7552 source:NTTCOM
220.231.124.0/23 Proxy-registered route object source:RADB
220.231.124.0/24 Proxy-registered route object source:RADB
220.231.125.0/24 Proxy-registered route object source:RADB
220.231.126.0/23 Proxy-registered route object source:RADB
220.231.126.0/24 Proxy-registered route object source:RADB
220.231.127.0/24 Proxy-registered route object source:RADB
2001:0dc8:3000::/48 VNNIC-AS-VN source:APNIC
2001:0dc8:c000::/48 VNNIC-AS-VN source:APNIC
2001:0dc8:c001::/48 VNNIC-AS-VN source:APNIC
2001:0dc8:d000::/48 VNNIC-AS-VN source:APNIC
2402:800:1130::/48 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:3000::/36 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:3100::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:3200::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:3300::/40 Prefix IPv6 Viettel-VN source:NTTCOM
............
............
............
2402:800:63e3::/48 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:63e4::/48 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:63e5::/48 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:7000::/36 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:7100::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:7200::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:7300::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8000::/36 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8100::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8100::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8110::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8120::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8130::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8140::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8200::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8200::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8210::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8220::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8230::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8240::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8300::/40 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8300::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8310::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8320::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8330::/44 Prefix IPv6 Viettel-VN source:NTTCOM
2402:800:8340::/44 Prefix IPv6 Viettel-VN source:NTTCOM

Cheers,

guy038

piranpiran · Jun 27, 2019, 11:06 PM

@guy038 Your dexterity with regex and S/R is a source of admiration to me. And, yes, that ASN listing is quite big. Admittedly my workstation’s spec is a great deal more powerful than that of the machine at your disposal.

The IPv4 CIDR element of that ASN comprises about 286k lines. Prior to 7.7.1 a simple click of integer sorting ascending on the relevant highlighted data would output a perfectly reliable result in ‘less than a second’ with other remaining data bunches nicely grouped for easy workflow assessment. It really did the job …but possibly not by strict npp design intention. BTW no crashes - ever.

Whittling those IPv4 CIDR lines down to unique values I do manually using techniques derived from long experience (sad to say). That workflow is made very much harder because rapid assessment (of the unnecessary ranges) is foiled by the inconsistent rendering of the 7.7.1 sort.

Despite your efforts I remain convinced the most ergonomic solution is likely to be a code fix - not one involving multiple regex S/R.

piranpiran · Jun 28, 2019, 10:17 AM

@guy038 said:

I just hope that these two regex S/R work nice, too, on your VERY BIG unordered ASN data list ;-))

Delighted to confirm that your hack (the first one) delivered a proper sort of the large listing (AS7552).

If you’re curious my kit took some 48sec compared to the inconsistent ‘results’ provided by 7.7.1 in (what would’ve normally have been) about a second.

Thank you @guy038 for providing that workaround to the native code’s flawed 7.7.1 …for which I await a fix.