Other thoughts regarding the "ReWrap Text to (Clipboard or 72) width" of the old "TextFX" plugin

mpheath

@guy038 Seems like regex Wrap versus the others using ReWrap. The test does not appear to be an equal comparison.

The test input files are UnWrapped. ReWrap would probably be for lines that are already Wrapped. Neither of the test files have a fully strict form of paragraphs. The license file is mostly paragraphs except for some lines.

GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

These 2 lines are supposed to remain as 2 lines, not 1 paragraph. So if treated with ReWrap, would probably result as 1 line.

Wrap can be done with this Lua code:

local function main()
    local maxLineLength = 72
    local line = 0
    local eol = ({'\r\n', '\r', '\n'})[editor.EOLMode + 1]

    editor:BeginUndoAction()

    while 1 do
        local lineLength = editor:LineLength(line)

        if lineLength < 1 then
            break
        end

        if lineLength > maxLineLength then
            local startPos = editor:PositionFromLine(line)
            local endPos = editor.LineEndPosition[line]
            local maxPos = startPos + maxLineLength
            local breakPos

            for i = startPos, endPos do
                local posAfter = editor:PositionAfter(i)

                if posAfter > (i + 1) then
                    maxPos = maxPos + posAfter - i - 1
                end

                if editor.CharAt[i] == 0x20 then
                    if breakPos and i > maxPos then
                        break
                    end

                    breakPos = i
                end
            end

            if breakPos and breakPos < endPos then
                editor:SetTargetRange(breakPos, breakPos + 1)
                editor:ReplaceTarget(eol)
            end
        end

        line = line + 1
    end

    editor:EndUndoAction()
end


main()

It replaces the space <= 72+ to break at with an EOL. Suggest to keep the original unwrapped copy as I have no UnWrap or ReWrap code.

Coises

This post is deleted!

Coises

@guy038 said:

Quite by chance, I came across a summary of the features of the old TextFX plugin, and among them, I took another look at the Rewrap feature.

First, note that the rewrap feature is generally VERY difficult to simulate as it heavily depends on current text :

How do we manage tables, sections, lists with bullets or numbers and possible leading indentation of the folded lines ?

Or do we need to join all lines of a text in a single very looooooooong line FIRST, before the wrapping process ?

As far as I can tell, the old TextFX didn’t handle any of the more complex cases (tables, lists, indentation). I have not been able to find user documentation. The closest reasonably accessible code I could find was this for the rewrap function, which has comments suggesting some fancier processing might be possible, but it doesn’t implement it.

Do any of the scripts or proposals offered so far do anything that can’t be accomplished with the native Split Lines function? Split Lines splits at the edge of the view if you have no vertical edges set, and at the last (or only) vertical edge listed if you do. (So if you list 72 80 it will split at 80; if you list 80 72 it will split at 72.)

What is the problem to be solved?

guy038

Hi, @mpheath, @coises and all,

@mpheath, I first copied the ORG_LF_ANSI.txt file as Lua_LF_ANSI.txt then I ran your last lua script against this file and the files are almost identical, except in the 3 locations below :

Then I copied again the ORG_LF_ANSI.txt file as Lua_CRLF_UTF-8.txt then I ran your last lua script against this file. This time, there are 5 differences, shown below :

Thus, as a conclusion, your Lua script is quite correct but the last regex solution :

SEARCH (?-s)^(.{1,78})(?:[\x20\r\n]*\z|\x20*\R|\x20+)

REPLACE \1\n ( for Unix files ) or \1\r\n ( for Windows files )

Followed with a Trim Trailing Space operation, seems slightly better !

Now, @coises, you said :

What is the problem to be solved?

Well, I just wanted to point out that, in particular cases, a simple wrapping operation may not give the expected display, from the user’s perspective ! In my opinion, it seems better to use the present @mpheath’s Lua script ( or an equivalent Python one ) or the regex solution above rather than the odd solution provided by the TextFX plugin.

Best Regards,

guy038

Oh…, I forgot to tell you, @mpheath, that I changed the line :

    local maxLineLength = 72

By this one :

    local maxLineLength = 78

For all the tests !

mpheath

@guy038 The Change History markers and indicators should be evaluated also as part of the tests. All green markers in the images is suspiciously odd. Perhaps all of the text was pasted and saved which caused that to occur? With Change History showing what changed, the view of what is better may change.

guy038

Hi, @mpheath and All,

Ah… I understand. Thus, I’ve decided to conduct a second round of tests. This time :

The files New_LF_ANSI.txt and New_CRLF_UTF-8.txt were copied from ORG_LF_ANSI.txt as Regex_LF_ANSI.txt and Regex_CRLF_UTF-8.txt
The files Lua_LF_ANSI.txt and Lua_CRLF_UTF-8.txt were copied from ORG_LF_ANSI.txt as Lua_2_LF_ANSI.txt and Lua_2_CRLF_UTF-8.txt

Then, I stopped and restarted N++ : thus, no more change history feature !

Finally, regarding these new created files :

I ran the regex replacement, followed by a Trim trailing Space operation, on files Regex_LF_ANSI.txt and Regex_CRLF_UTF-8.txt
I ran your Lua script on files Lua_2_LF_ANSI.txt and Lua_2_CRLF_UTF-8.txt

So, these four files showed an orange change history color. As you can see on the screenshots, the regex process acted on all lines, but some empty line endings, and your lua script just acted on lines which need wrapping !

Regex process :

Lua process :

However, the comparisons between the regex process and the Lua process shows the same number of differences ( 3 for ANSI files and 5 for UTF-8 files )

Between ANSI files :

Between UTF-8 files

Best Regards,

guy038

mpheath

@guy038 The testing looks better.

If the Change History -> Show in text option is checked then can see changes to a character for more detail. No need to retest and repost images to show the indicators.

A change of the regex pattern to:

Find:

(?-s)^(?=.{79}).{1,78}\K\x20(?!\R)

Replace:

\r\n

and then press the Replace All button.

It does replace less as it does not capture and so only the \x20 space is matched. Unfortunately, it will read the line to length of 79 and then read the line again if line is 79 to continue the pattern so might be slower then your previous tested pattern. The (?=.{79}) is needed to ensure shorter lines are not matched for wrapping. The (?!\R) is probably optional to have in the pattern as it is limited to detecting trailing spaces. Since the pattern uses \K, then the Replace button is not expected to function properly to step through the replacements.

The text comparison in my testing shows the regex patterns achieve equal results which seems to be fortunate. That test was done on license.txt.

guy038

Hi, @mpheath and All,

@mpheath, from your advice, I finally chose the following process :

First, run a Edit > Blank Operations > TAB to Space option
Then run the regex replacement below, where C represents the maximum number of chars after you want a line-break :
- SEARCH [\x20\r\n]*\z|(?-s)^(?=.{C+1,}).{1,C}\K(?:\x20*\R|\x20+)
- REPLACE \r\n for Windows files or \n for Unix files
Finally, run a Edit > Blank operations > Trim Trailing Space option

Remarks :

I began the regex with [\x20\r\n]*\z which will replace possible mixed range of spaces and new lines, at the very end of current file, with a single line-ending
I used the (?-s)^(?=.{C+1,}) ( instead of the (?-s)^(?=.{C+1}) as it seems slightly faster within huge files !
Compare to your regex, I still wanted to consider all spaces, possibly followed with a line-break. So, the part .{1,C}\K(?:\x20*\R|\x20+)
Note that the final Trim Trailing Space action is necessary because of this specific case with a splitting after column 40

12345678901234567890123456789012345     ABCDE

12345678901234567890123456789012345     
ABCDE

Now, let’s try it out with a concrete example :

Double-click on https://www.gutenberg.org/cache/epub/2701/pg2701.txt => The text of the novel Moby Dick or The Whale should appear
Go to the second occurrence of the string CHAPTER 1. Loomings.
Select from this line to the line END OF THE PROJECT GUTENBERG EBOOK MOBY DICK; OR, THE WHALE, excluded
Paste it in a N++ new tab and name it Moby_ORG.txt

So, the Moby.ORG.txt file should begin and end with :


Call me Ishmael. Some years ago—never mind how long precisely—having
little or no money in my purse, and nothing particular to interest me
on shore, I thought I would sail about a little and see the watery part
of the world. It is a way I have of driving off the spleen and
regulating the circulation. Whenever I find myself growing grim about
...
...
...
...
unharming sharks, they glided by as if with padlocks on their mouths;
the savage sea-hawks sailed with sheathed beaks. On the second day, a
sail drew near, nearer, and picked me up at last. It was the
devious-cruising Rachel, that in her retracing search after her missing
children, only found another orphan.

In other words :

Line 1 EMPTY
Line 2 FIRST line of text
Line 20,707 LAST line of text
Line 20,708 = \z

As this file seems already wrapped to 71 chars, I decided to do, FIRST, an intelligent un-wrapping !

I verified that there was no tabulation, nor any trailing space in the Moby_ORG.txt file
Then I noticed that 43 lines began with one or two space characters, without any real utility. So, I ran the Edit > Blanck operations > Trim Trailing Space option
To get a correct un-wrapping process, we need to use the following regex S/R :
- SEARCH (?-s).+\K\R(?=.+) ( Do not insert a ^ character between the parts (?-s) and .+ ) of this regex ! )
- REPLACE \x20
Finally, I saved the changes and renamed the Moby_ORG.txt as Moby_To_Wrap.txt

So, the Moby_To_Wrap.txt file begins and ends as below :

Call me Ishmael. Some years ago—never mind how long precisely—having little ...

There now is your insular city of the Manhattoes, belted round by wharves as ...

Circumambulate the city of a dreamy Sabbath afternoon. Go from Corlears Hook ...
...
...
...
...
“AND I ONLY AM ESCAPED ALONE TO TELL THEE” Job.

The drama’s done. Why then here does any one step forth?—Because one ...

It so chanced, that after the Parsee’s disappearance, I was he whom ...

In other words :

Line 1 EMPTY
Line 2 FIRST line of text
Line 4,884 LAST line of text
Line 4,885 = \z

The second line of your script is always :

local maxLineLength = 78

And the Settings > Preferences... Margins/Border/Edge > Vertival Edge Settings only contains the 78 value

Then I copied the Moby_ORG.txt file three times, as Moby_Regex.txt, Moby_Lua.txt and Moby_Ctrl-I.txt and I applied :

Your script against the Moby_Lua.txt file,
The Edit > Line Operations > Slipt Lines option against the Moby_Ctrl-I.txt
The following regex S/R against the Moby_Regex.txt
- SEARCH [\x20\r\n]*\z|(?-s)^(?=.{79,}).{1,78}\K(?:\x20*\R|\x20+)
- REPLACE \r\n

Upon comparison, it turns out that these three files are completely different ! Compared to the Moby_Regex.txt file , I noticed that your Moby_Lua.txt sometimes grasps :

More than it should, as, for example, in the lines :

“Who-e debel you?”—he at last said—“you no speak-e, dam-me, I kill-e.” And so saying with a wrapping after 84 chars

“Oh, I don’t mind ’em, sir,” said Archy; “I knew it all before now. Didn’t I hear ’em with a wrapping after 85 chars

Less than it should, as, for example, in lines :

the Stoics to enable you to grin and bear it. But even this wears off in, with a wrapping after 72 chars

knowingly, would thereby render yourself liable to a criminal, with a wrapping after 61 chars

Not important, though. Just in case you would work on your script a little more !

Now, using this long text ( the Moby_To Rewrap.txt file ), I confirm that the regex, below, exactly mimics the Ctrl + I feature :

SEARCH (?-s)^(?=.{78})(?:.{1,76}\x20|.{77})\K

REPLACE \r\n for Windows files or \n for Unix files

However, note that the Edit > Line Operations > Split lines operation leaves a trailing space on all lines of a paragrah but the last one !

I think that further investigation would be useless and I’ll personally just use the method, shown at the beginning of this post, when I need to wrap any text !

Best Regards,

guy038

Oh, I almost forgot something important :

When you select some text or all contents of a file and execute the Edit > Line Operations > Split Lines option :

The View > Word wrap must be OFF
The View > Show Symbol must be OFF

Alan Kilborn

@guy038 said:

just use the method, shown at the beginning of this post, when I need to wrap any text

For clarity for future readers of this thread, which method is this one?

When you select some text or all contents of a file and execute the Edit > Line Operations > Split Lines option :

The View > Word wrap must be OFF

The View > Show Symbol must be OFF

And, because I’m currently away from my Notepad++ and can’t try it myself, why are these "must be OFF"s?

guy038

Hello, @alan-kilborn,

You said :

For clarity for future readers of this thread, which method is this one?

I was referring to these steps :

First, run a Edit > Blank Operations > TAB to Space option
Then run the regex replacement below, where C represents the maximum number of chars after you want a line-break :
- SEARCH [\x20\r\n]*\z|(?-s)^(?=.{C+1,}).{1,C}\K(?:\x20*\R|\x20+)
- REPLACE \r\n for Windows files or \n for Unix files
Finally, run a Edit > Blank operations > Trim Trailing Space option

You then said :

And, because I’m currently away from my Notepad++ and can’t try it myself, why are these "must be OFF"s?

I must apologize because I was mistaken ! I did some other tests and it happens that the status of the View > Word wrap does not matter

But I confirm that showing the line-endings chars breaks the correct spliting of text when using the Edit > Line Operations > Split Lines option

STR :

In Settings > Preferences... > Margins/Border/Edge > Vertical Edge Settings insert the value 78 , ONLY
For clarity, just uncheck the View > Word wrap option and the View > Show Symbol option
Paste the beginning of the License.txt file, below, in a new tab :

COPYING -- Describes the terms under which Notepad++ is distributed.
A copy of the GNU GPL is appended to this file.

IMPORTANT NOTEPAD++ LICENSE TERMS

Copyright (C)2021 Don HO <don.h@free.fr>.  This program is free software; you may redistribute and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; Version 3 with the clarifications and exceptions described below. This guarantees your right to use, modify, and redistribute this software under certain conditions.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

******************************************************************


GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

Preamble
The GNU General Public License is a free, copyleft license for software and other kinds of works.

The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too.

Now, in the new tab, select all this text with Ctrl + A
Run the Edit > Line Operations > Slipt Lines or Ctrl + I option

=> You should get the text below :

COPYING -- Describes the terms under which Notepad++ is distributed.
A copy of the GNU GPL is appended to this file.

IMPORTANT NOTEPAD++ LICENSE TERMS

Copyright (C)2021 Don HO <don.h@free.fr>.  This program is free software; 
you may redistribute and/or modify it under the terms of the GNU General 
Public License as published by the Free Software Foundation; Version 3 with 
the clarifications and exceptions described below. This guarantees your 
right to use, modify, and redistribute this software under certain 
conditions.

This program is distributed in the hope that it will be useful, but WITHOUT 
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for 
more details.

******************************************************************


GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>

Everyone is permitted to copy and distribute verbatim copies of this license 
document, but changing it is not allowed.

Preamble
The GNU General Public License is a free, copyleft license for software and 
other kinds of works.

The licenses for most software and other practical works are designed to 
take away your freedom to share and change the works. By contrast, the GNU 
General Public License is intended to guarantee your freedom to share and 
change all versions of a program--to make sure it remains free software for 
all its users. We, the Free Software Foundation, use the GNU General Public 
License for most of our software; it applies also to any other work released 
this way by its authors. You can apply it to your programs, too.

Now, undo, with Ctrl + Z, the last action in order to return to the unwrapped text again
Click on the ¶ icon or run the View > Show Symbols > Show All Characters option
RE-select all contents of the new tab
Run again the Edit > Line Operations > Slipt Lines or Ctrl + I option

=> This time, you should get this text :

COPYING -- Describes the terms under which Notepad++ is distributed.
A copy of the GNU GPL is appended to this file.

IMPORTANT NOTEPAD++ LICENSE TERMS

Copyright (C)2021 Don HO <don.h@free.fr>.  This program is free software; 
you may redistribute and/or modify it under the terms of the GNU 
General 
Public License as published by the Free Software Foundation; Version 3 
with 
the clarifications and exceptions described below. This guarantees your 
right to use, modify, and redistribute this software under certain 
conditions.

This program is distributed in the hope that it will be useful, but WITHOUT 
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or 
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License 
for 
more details.

******************************************************************


GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007

Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>

Everyone is permitted to copy and distribute verbatim copies of this license 
document, but changing it is not allowed.

Preamble
The GNU General Public License is a free, copyleft license for software and 
other kinds of works.

The licenses for most software and other practical works are designed to 
take away your freedom to share and change the works. By contrast, the 
GNU 
General Public License is intended to guarantee your freedom to share 
and 
change all versions of a program--to make sure it remains free software 
for 
all its users. We, the Free Software Foundation, use the GNU General 
Public 
License for most of our software; it applies also to any other work 
released 
this way by its authors. You can apply it to your programs, too.

And, obviously, these two texts are quite different !

Test done with a portable N++ v8.9.8 - 64 bits installation !

Best Regards

guy038