Unicode 'ÿ' , problem converting to Hex 'FF'
-
@Mark-Olson I did not know that about VSCode, thanks, Ill look into it. Mark.
-
Hello, @alan_kilborn and All,
So, here is the contents of this macro :
<Macro name="Surrogates Pairs in Selection" Ctrl="no" Alt="no" Shift="no" Key="0"> <Action type="3" message="1700" wParam="0" lParam="0" sParam="" /> <Action type="3" message="1601" wParam="0" lParam="0" sParam="(?-i)\\x\{(10|[[:xdigit:]])[[:xdigit:]]{4}" /> <Action type="3" message="1625" wParam="0" lParam="2" sParam="" /> <Action type="3" message="1602" wParam="0" lParam="0" sParam="$0\x1F" /> <Action type="3" message="1702" wParam="0" lParam="640" sParam="" /> <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" /> <Action type="3" message="1700" wParam="0" lParam="0" sParam="" /> <Action type="3" message="1601" wParam="0" lParam="0" sParam="(?i)(?:(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(A)|(B)|(C)|(D)|(E)|(F)|(10))(?=[[:xdigit:]]{4}\x1F\})|(?:(0)|(1)|(2)|(3)|(4)|(5)|(6)|(7)|(8)|(9)|(A)|(B)|(C)|(D)|(E)|(F))(?=[[:xdigit:]]{0,3}\x1F\})" /> <Action type="3" message="1625" wParam="0" lParam="2" sParam="" /> <Action type="3" message="1602" wParam="0" lParam="0" sParam="(?{1}0000)(?{2}0001)(?{3}0010)(?{4}0011)(?{5}0100)(?{6}0101)(?{7}0110)(?{8}0111)(?{9}1000)(?{10}1001)(?{11}1010)(?{12}1011)(?{13}1100)(?{14}1101)(?{15}1110)(?{16}1111)(?{17}0000)(?{18}0001)(?{19}0010)(?{20}0011)(?{21}0100)(?{22}0101)(?{23}0110)(?{24}0111)(?{25}1000)(?{26}1001)(?{27}1010)(?{28}1011)(?{29}1100)(?{30}1101)(?{31}1110)(?{32}1111)" /> <Action type="3" message="1702" wParam="0" lParam="640" sParam="" /> <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" /> <Action type="3" message="1700" wParam="0" lParam="0" sParam="" /> <Action type="3" message="1601" wParam="0" lParam="0" sParam="([01]{10})([01]{10})(?=\x1F)" /> <Action type="3" message="1625" wParam="0" lParam="2" sParam="" /> <Action type="3" message="1602" wParam="0" lParam="0" sParam="110110\1\x1F}\\x{110111\2" /> <Action type="3" message="1702" wParam="0" lParam="640" sParam="" /> <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" /> <Action type="3" message="1700" wParam="0" lParam="0" sParam="" /> <Action type="3" message="1601" wParam="0" lParam="0" sParam="(?:(0000)|(0001)|(0010)|(0011)|(0100)|(0101)|(0110)|(0111)|(1000)|(1001)|(1010)|(1011)|(1100)|(1101)|(1110)|(1111))(?=[[:xdigit:]]*\x1F\})|\x1F" /> <Action type="3" message="1625" wParam="0" lParam="2" sParam="" /> <Action type="3" message="1602" wParam="0" lParam="0" sParam="(?{1}0)(?{2}1)(?{3}2)(?{4}3)(?{5}4)(?{6}5)(?{7}6)(?{8}7)(?{9}8)(?{10}9)(?11A)(?12B)(?13C)(?14D)(?15E)(?16F)" /> <Action type="3" message="1702" wParam="0" lParam="640" sParam="" /> <Action type="3" message="1701" wParam="0" lParam="1609" sParam="" /> </Macro>
How this macro works :
-
You select a true Unicode value of a character over the
BMP:\x{#####}or\x{######}, with5or6hexadecimal digits -
You run the macro
-
You get its surrogate pair :
\x{####}\x{####}
Note that your normal selection can encompass several
\x{#####}syntaxes in one go. For example, after selection of the text, below and execution of the macro :\x{10000} is the FIRST char over the BMP This SECOND char \x{14A9B} is concerned by this macro, too A THIRD char \x{14A9D}, very closed to the PREVIOUS one A FOURTH example, at the VERY END of the UNICODE range = \x{10FFFF}This text will be changed as :
\x{D800}\x{DC00} is the FIRST char over the BMP This SECOND char \x{D812}\x{DE9B} is concerned by this macro, too A THIRD char \x{D812}\x{DE9D}, very closed to the PREVIOUS one A FOURTH example, at the VERY END of the UNICODE range = \x{DBFF}\x{DFFF}
Now, Alan, I will concede that, for a single char, this macro is not very useful as you can, instead, just select this character over the `BMP, open the search dialog and search for all occurrences of this specific character !
But, let’s suppose you would scan your files for those containing, let’s say, the Unicode Emoticons range of chars, which lays between the
\x{1F600}and the\x{1F64F}characters. Refer to :http://www.unicode.org/charts/PDF/U1F600.pdf
And which is reported, below :
😀 \x{1F600} = \x{D83D}\x{DE00}
😁 \x{1F601} = \x{D83D}\x{DE01}
😂 \x{1F602} = \x{D83D}\x{DE02}
😃 \x{1F603} = \x{D83D}\x{DE03}
😄 \x{1F604} = \x{D83D}\x{DE04}
😅 \x{1F605} = \x{D83D}\x{DE05}
😆 \x{1F606} = \x{D83D}\x{DE06}
😇 \x{1F607} = \x{D83D}\x{DE07}
😈 \x{1F608} = \x{D83D}\x{DE08}
😉 \x{1F609} = \x{D83D}\x{DE09}
😊 \x{1F60A} = \x{D83D}\x{DE0A}
😋 \x{1F60B} = \x{D83D}\x{DE0B}
😌 \x{1F60C} = \x{D83D}\x{DE0C}
😍 \x{1F60D} = \x{D83D}\x{DE0D}
😎 \x{1F60E} = \x{D83D}\x{DE0E}
😏 \x{1F60F} = \x{D83D}\x{DE0F}
😐 \x{1F610} = \x{D83D}\x{DE10}
😑 \x{1F611} = \x{D83D}\x{DE11}
😒 \x{1F612} = \x{D83D}\x{DE12}
😓 \x{1F613} = \x{D83D}\x{DE13}
😔 \x{1F614} = \x{D83D}\x{DE14}
😕 \x{1F615} = \x{D83D}\x{DE15}
😖 \x{1F616} = \x{D83D}\x{DE16}
😗 \x{1F617} = \x{D83D}\x{DE17}
😘 \x{1F618} = \x{D83D}\x{DE18}
😙 \x{1F619} = \x{D83D}\x{DE19}
😚 \x{1F61A} = \x{D83D}\x{DE1A}
😛 \x{1F61B} = \x{D83D}\x{DE1B}
😜 \x{1F61C} = \x{D83D}\x{DE1C}
😝 \x{1F61D} = \x{D83D}\x{DE1D}
😞 \x{1F61E} = \x{D83D}\x{DE1E}
😟 \x{1F61F} = \x{D83D}\x{DE1F}
😠 \x{1F620} = \x{D83D}\x{DE20}
😡 \x{1F621} = \x{D83D}\x{DE21}
😢 \x{1F622} = \x{D83D}\x{DE22}
😣 \x{1F623} = \x{D83D}\x{DE23}
😤 \x{1F624} = \x{D83D}\x{DE24}
😥 \x{1F625} = \x{D83D}\x{DE25}
😦 \x{1F626} = \x{D83D}\x{DE26}
😧 \x{1F627} = \x{D83D}\x{DE27}
😨 \x{1F628} = \x{D83D}\x{DE28}
😩 \x{1F629} = \x{D83D}\x{DE29}
😪 \x{1F62A} = \x{D83D}\x{DE2A}
😫 \x{1F62B} = \x{D83D}\x{DE2B}
😬 \x{1F62C} = \x{D83D}\x{DE2C}
😭 \x{1F62D} = \x{D83D}\x{DE2D}
😮 \x{1F62E} = \x{D83D}\x{DE2E}
😯 \x{1F62F} = \x{D83D}\x{DE2F}
😰 \x{1F630} = \x{D83D}\x{DE30}
😱 \x{1F631} = \x{D83D}\x{DE31}
😲 \x{1F632} = \x{D83D}\x{DE32}
😳 \x{1F633} = \x{D83D}\x{DE33}
😴 \x{1F634} = \x{D83D}\x{DE34}
😵 \x{1F635} = \x{D83D}\x{DE35}
😶 \x{1F636} = \x{D83D}\x{DE36}
😷 \x{1F637} = \x{D83D}\x{DE37}
😸 \x{1F638} = \x{D83D}\x{DE38}
😹 \x{1F639} = \x{D83D}\x{DE39}
😺 \x{1F63A} = \x{D83D}\x{DE3A}
😻 \x{1F63B} = \x{D83D}\x{DE3B}
😼 \x{1F63C} = \x{D83D}\x{DE3C}
😽 \x{1F63D} = \x{D83D}\x{DE3D}
😾 \x{1F63E} = \x{D83D}\x{DE3E}
😿 \x{1F63F} = \x{D83D}\x{DE3F}
🙀 \x{1F640} = \x{D83D}\x{DE40}
🙁 \x{1F641} = \x{D83D}\x{DE41}
🙂 \x{1F642} = \x{D83D}\x{DE42}
🙃 \x{1F643} = \x{D83D}\x{DE43}
🙄 \x{1F644} = \x{D83D}\x{DE44}
🙅 \x{1F645} = \x{D83D}\x{DE45}
🙆 \x{1F646} = \x{D83D}\x{DE46}
🙇 \x{1F647} = \x{D83D}\x{DE47}
🙈 \x{1F648} = \x{D83D}\x{DE48}
🙉 \x{1F649} = \x{D83D}\x{DE49}
🙊 \x{1F64A} = \x{D83D}\x{DE4A}
🙋 \x{1F64B} = \x{D83D}\x{DE4B}
🙌 \x{1F64C} = \x{D83D}\x{DE4C}
🙍 \x{1F64D} = \x{D83D}\x{DE4D}
🙎 \x{1F64E} = \x{D83D}\x{DE4E}
🙏 \x{1F64F} = \x{D83D}\x{DE4F}In this case, knowing the first and last
surrogatevalues of the Emoticons range, which are\x{D83D}\x{DE00}and\x{D83D}\x{DE4F}, it’s easier to get the right regex to search for, which is\x{D83D}[\x{DE00}-\x{DE4F}]!
And, indeed :
-
Open a new tab
-
Paste the
80characters of the Emoticons range, with their associated value, above, in this new tab -
Open the Mark dialog (
Ctrl + M) -
Tick the
Wrap aroundbox and select theRegular expressionmode -
Enter the regex
\x{D83D}[\x{DE00}-\x{DE4F}]in the Search zone -
Click on the
Mark Allbutton
=> The
80characters, of this Unicode range, should be marked in red-orange, all at once !Best Regards,
guy038
-
-
@guy038 Wow - Thanks for this macro!
-
@LanceMarchetti said in Unicode 'ÿ' , problem converting to Hex 'FF':
“
NOTE: Using a space ‘20’ to replace a NUL ‘00’ will still result in a 32x32 Gif image,”
Correction: I just noticed that the GIF file format repeats the image dimension block later on in the binary, just before the image data begins.
So I tried using Hex 20 20 20 20 in the second dimension sector as well. The result was INDEED a 8224x8224 image!
Changing it back to 20 00 20 00 resulted in a 32x32 image.Just wanted to set that straight. Thanks.
-
I was surprised that no one had suggested submitting a bug report to the Converter Plugin repo’s issue list regarding the
ÿportion, because even though most of your request is abusing a text editor to edit a non-text file, Notepad++'s Converter Plugin should be able to handle convertingÿin a valid ANSI file (win1252 or ISO-8859-1 or similar codepage).But when I went looking through open and closed issues, I found it’s actually already been reported multiple times – most recently, in #11… and #11 even came with a fix in PR#16, which was rejected because Don didn’t have the Steps to Reproduce.
Based on this conversation, I was able to come up with a minimal Steps to Reproduce, and asked Don to re-open that issue, and hopefully he will then see whether the fix #16 will work, or will come up with one that will.
The right answer to your full query is “Notepad++ is not a binary-file editor, so you need to accept hacky suggestions”. But
ÿshould be able to be handled by the Converter plugin, so I poked theÿissue. -
I didn’t look super closely at it (due to limited interest), but I’d guess this line/block is the cause of the problem:
At least it seems suspicious. 0xFF can look a lot like -1. :-)
-
@Alan-Kilborn said in Unicode 'ÿ' , problem converting to Hex 'FF':
At least it seems suspicious. 0xFF can look a lot like -1. :-)
Given that PR#16 changed the
getCharreturn value, which is what that -1 is comparing against, I’d say you’re probably right. -
@PeterJones said in Unicode 'ÿ' , problem converting to Hex 'FF':
Given that PR#16 changed the getChar return value, which is what that -1 is comparing against, I’d say you’re probably right.
I didn’t look beyond the current code.
-
@PeterJones Thanks Peter. Appreciated. I also emailed Don last week on the ÿ issue. But have not heard back yet. So I’m grateful that you clubbed in to hopefully get it sorted.
I’m not a programmer as stated before, so i think you Guys are obviously on the right track with ‘-1’ because Mistral AI identified it also as such.
/chat.mistral.ai/chat/ec2b9a74-2da7-4bda-bfb3-e2b8681c435f/
I see that the provided code snippet is part of the Ascii to Hex plugin for Notepad++. Based on my analysis, the issue likely arises from the fact that the ‘ÿ’ character (Alt+0255) corresponds to the unsigned char value 255, which is interpreted as a negative value (-1) when stored in a signed char. The code then treats this negative value as an error and refuses to convert the whole block of highlighted text.
To fix this issue, you can modify the code to use unsigned characters instead of signed characters. Here’s an updated version of the relevant part of the code:
void ascii2hex(bool insertSpace, bool isMaj, size_t nbCharPerLine) { // ... (previous code remains unchanged) unsigned char *pDestText = new unsigned char[textLen*(inc+eolNbChar)+1]; // ... (previous code remains unchanged) const char *format = ""; int val = static_cast<unsigned char>(selText.getChar(i)); if (val == -1) { delete [] pDestText; return; } // ... (remaining code) }By changing the
pDestTexttype tounsigned charand casting the result ofselText.getChar(i)tounsigned char, the code should now properly handle the ‘ÿ’ character and convert it to ‘FF’ in hexadecimal. -
@LanceMarchetti said in Unicode 'ÿ' , problem converting to Hex 'FF':
int val = static_cast<unsigned char>(selText.getChar(i));I’m afraid this type cast will corrupt data if any character ordinal is greater than
0xFF, e.g.ġ. See, for example:selText.getCharalready returns anint, which is 32-bits wide, so more than enough for any ASCII or Unicode ordinal.The real issue is the declaration of
SelectedString::_straschar *, which limits every representable “byte” to a maximum ordinal value of0x7F. It should be the genericTCHAR *type, which becomeswchar_t *when the_UNICODEcompile-time definition is set (as it always has been ever since Notepad++ became a Unicode application about 17 years ago). Awchar_tis 16 bits, which would accommodate “extended” 8-bit ASCII as well as the East Asian double-byte character sets that remain popular with some users.
