How to normalize fancy Unicode text back to regular text?

[Alan Kilborn]

@guy038 said in How to normalize fancy Unicode text back to regular text?:

I advice you to simply use this on-line tool

OP has already said that such usage is undesirable; well, I think that is what he is saying with:

Otherwise we are depending on other sources like those websites who support that function and that a disadvantage for npp and npp users

---

Isn’t the true answer what @PeterJones has shown is possible, with a script?
Such a script could act on selected text when the script is run, and replace that text with the normalized text…pretty simple concept.

[Coises]

@mkupper said in How to normalize fancy Unicode text back to regular text?:

Some characters do change. Copy/paste the following into a UTF-8 encoded tab or file. It should be the same as when you see here on the forums.

I stand corrected. I did not at all expect that to happen. It’s my understanding of “convert to ANSI” that is confused. I apologize.

Very strange:

Open Notepad++, convert empty tab to UTF-8, copy your text, paste into tab, I see all the characters.

Copy your text, open Notepad++, convert empty tab to UTF-8, paste into tab, I see only ASCII characters.

I have no idea what is going on here.

[Mark Olson]

@Alan-Kilborn said in How to normalize fancy Unicode text back to regular text?:

Such a script could act on selected text when the script is run, and replace that text with the normalized text…pretty simple concept.

Might as well just make the script now, save others time.

As noted in the docstring of the code, the most obvious difference between NFKD and NFKC seems to be treatment of characters with combining diacritics or umlauts or what have you. Which form is better seems really context-dependent to me; if you’re sorting text, you probably want ö to be an o and then an umlaut (so that ö sorts after o and before p as expected), but if you’re doing regular expression search, you might prefer it to be a single character.

'''
requires PythonScript v3 or higher: https://github.com/bruderstein/PythonScript
ref: https://community.notepad-plus-plus.org/topic/25285/how-to-normalize-fancy-unicode-text-back-to-regular-text/17
docs: https://docs.python.org/3.10/library/unicodedata.html
'''
import unicodedata
from Npp import *

def normalize(text):
    '''
    NFKC stands for normalization form compatibility decomposition
        with subsequent canonical composition.
    NFKD works similarly AFAIK; it may be a bit faster, but it has some weird 
        behaviors like breaking ö into two characters: ASCII "o" and then ̈
        whereas NFKC combines those two into a single character.
    '''
    return unicodedata.normalize('NFKC', text)

selstart = editor.getSelectionStart()
selend = editor.getSelectionEnd()

if selstart == selend:
    text = editor.getText()
    editor.setText(normalize(text))
else:
    text = editor.getSelText()
    editor.replaceSel(normalize(text))

[guy038]

Hi, @alan-kilborn,

I completely agree with your last assumption and that why I had already upvoted @peterjones’s post and I now upvote to @mark-olson’s solution too !

BR

guy038

[Dean-Corso]

Hi guys,

thanks again for your help. Really nice from you all.

@PeterJones

Thanks for hint about the python script versions. I did download the latest pre version as you but could not make the same steps like you did to enter your example lines. Got some errors trying to exec the print command (getting expand error on for statement etc). Just did enter same as you. Maybe some space issue or something not sure. But good to know that I needed to use a higher python 3x version so I was still using the older 2x version.

@Mark-Olson

Thank you for that example script. I tried that one and it seems to work. Great! The results are very good for me and its working for some of those different symbol styles (not all) to get a rid of those symbol text at all or some mixed plain text with symbol text etc. I mean the script works same like those few websites I found to normalize the symbol text to plain text. That’s very good and I don’t need to use those websites anymore and that was one of my goals. Would be good when npp could make a build in function for that in any future releases if possible.

[PeterJones]

@Dean-Corso said in How to normalize fancy Unicode text back to regular text?:

Got some errors trying to exec the print command (getting expand error on for statement etc). Just did enter same as you. Maybe some space issue or something not sure.

If you copy/pasted the PythonScript console results (including the version information) like I did above, I bet someone could tell you what happened

[Dean-Corso]

@PeterJones

Ok I tried again and now I get this out…

Python 3.12.1 (tags/v3.12.1:2305ca5, Dec  7 2023, 22:03:25) [MSC v.1937 64 bit (AMD64)]
Initialisation took 204ms
Ready.
>>> import unicodedata
>>> strings = [   '𝖙𝖍𝖚𝖌 𝖑𝖎𝖋𝖊',   '𝓽𝓱𝓾𝓰 𝓵𝓲𝓯𝓮',   '𝓉𝒽𝓊𝑔 𝓁𝒾𝒻𝑒',   '𝕥𝕙𝕦𝕘 𝕝𝕚𝕗𝕖',   'ｔｈｕｇ ｌｉｆｅ', '𝘏𝘦𝘭𝘭𝘰 𝘕𝘰𝘵𝘦𝘱𝘢𝘥 𝘱𝘭𝘶𝘴 𝘱𝘭𝘶𝘴 𝘤𝘰𝘮𝘮𝘶𝘯𝘪𝘵𝘺', '𝙃𝙚𝙡𝙡𝙤 𝙉𝙤𝙩𝙚𝙥𝙖𝙙 𝙥𝙡𝙪𝙨 𝙥𝙡𝙪𝙨 𝙘𝙤𝙢𝙢𝙪𝙣𝙞𝙩𝙮']
>>> for x in strings:
...   print(unicodedata.normalize( 'NFKC', x), x)

…but don’t see the printed output like you have. Did I miss anything to enter in this case?

PS: About that error before, I see I forgot to enter another white space before last print command.

[PeterJones]

@Dean-Corso ,

If your PythonScript console prompt is still ... instead of >>>, you will need to enter a blank line (no whitespace) to tell the console to end the loop. It won’t run the loop until you do.

[Screen White]

Hi @Dean-Corso, @PeterJones, @Mark-Olson, @guy038, and everyone,

First, thank you for this excellent discussion. I particularly appreciate the detailed investigation into the Mathematical Alphanumeric Symbols block, the limitations of regular expressions in Notepad++, and the eventual working PythonScript solution.

Mark Olson’s NFKC-based script already provides a good solution to the original question, and it is encouraging to see that Dean confirmed it works for many of the styled Unicode examples.

I would like to add a few practical considerations for anyone using this approach with larger documents, mixed-language text, or Unicode strings copied from social media.

1. A safer normalization workflow for selected text

One potential improvement is to make normalization explicitly selection-based, especially when working with documents containing code, mathematical symbols, or multilingual text.

NFKC is not merely a visual font converter. It can also transform compatibility characters whose distinctions may matter in certain contexts.

For users who want to normalize only copied fancy text, I would suggest a script that modifies the selected region and leaves the rest of the document untouched.

For PythonScript 3.x:

import unicodedata
from Npp import *

selected = editor.getSelText()

if not selected:
    console.write(
        "Please select the text to normalize.\n"
    )
else:
    normalized = unicodedata.normalize(
        "NFKC", selected
    )

    if normalized == selected:
        console.write(
            "No changes under NFKC normalization.\n"
        )
    else:
        editor.beginUndoAction()

        try:
            editor.replaceSel(normalized)
        finally:
            editor.endUndoAction()

        console.write(
            "NFKC normalization completed. "
            "Use Ctrl+Z to undo.\n"
        )

This version deliberately requires a normal text selection instead of automatically modifying the entire document when nothing is selected.

It also groups the replacement into a single undoable action.

For whole-document conversion, users can simply select all text first.

This example is intended for a standard, single text selection; rectangular or multiple-selection workflows would need additional handling.

2. NFKC handles more Unicode styles than manually maintained regex mappings

A major advantage of Unicode compatibility normalization is that it relies on the Unicode character database instead of a manually assembled collection of replacement expressions.

Consider these examples:

𝘏𝘦𝘭𝘭𝘰 → Hello

𝙃𝙚𝙡𝙡𝙤 → Hello

𝐇𝐞𝐥𝐥𝐨 → Hello

Ⓗⓔⓛⓛⓞ → Hello

ｈｅｌｌｏ → hello

These examples can be normalized using the same NFKC operation.

This avoids maintaining separate regex substitutions for every mathematical bold, italic, sans-serif, script, or fullwidth alphabet.

It also avoids some of the complexity discussed earlier concerning supplementary-plane characters and UTF-16 surrogate pairs.

However, an important qualification is that NFKC does not guarantee conversion of every visually decorative character into ASCII.

3. Why some fancy text does not return to normal

This is perhaps the most important limitation to explain to users.

There are different mechanisms for producing what people commonly call “fancy fonts.”

Type A: Compatibility characters

Examples include mathematical bold, italic, Fraktur, and fullwidth letters.

Many of these characters have Unicode compatibility mappings to ordinary letters.

NFKC can normalize them.

Type B: Combining-mark decorations

Consider:

H̸e̸l̸l̸o̸

This text contains ordinary letters with additional combining overlay marks.

NFKC does not simply delete those marks.

Removing them requires an additional, explicitly defined transformation, and that transformation may also remove meaningful linguistic information.

Type C: Visually confusable characters

Some Latin, Greek, and Cyrillic letters look similar but represent different Unicode characters.

For example, a string may contain a Cyrillic character that visually resembles a Latin letter.

NFKC does not automatically treat all such characters as equivalent.

This is a separate problem involving script identification and Unicode confusables.

Type D: Decorative symbols and emoji

Some generators add symbols, enclosing characters, emoji, or zero-width joiner sequences.

There is no general Unicode normalization operation that can reconstruct the author’s original text from every possible decoration.

Consequently, I would avoid promising an “all fancy Unicode to ASCII” conversion.

A more accurate description is “Unicode compatibility normalization, with optional application-specific cleanup.”

4. Distinguish normalization from transliteration and encoding conversion

Another important distinction raised earlier in this thread is the difference between converting document encoding and changing the actual characters.

These operations solve different problems.

Encoding conversion

Changes how text is represented in storage, such as converting between UTF-8 and a legacy code page.

Unicode normalization

Transforms certain character sequences according to defined Unicode equivalence rules.

Transliteration

Converts text between writing systems or provides approximate representations using another script.

Diacritic removal

Removes selected combining marks or accent information.

These operations should not be treated as interchangeable.

For example, the accented word café remains café under NFKC.

If an application specifically requires ASCII-only output, that requires an additional policy for unsupported letters, marks, symbols, and scripts.

Silently replacing unsupported characters with question marks would usually be undesirable for text recovery.

5. Fancy Unicode generators can help create regression tests

Another practical suggestion is to use Unicode text generators to build a representative collection of inputs for testing normalization scripts.

Instead of testing only one italic alphabet, we can generate several styles from the same original phrase and check which ones normalize correctly.

For example, start with:

Hello Notepad

Generate mathematical bold, italic, script, double-struck, Fraktur, circled, and decorated variants.

Then compare the outputs after NFKC normalization.

A practical Unicode text generator that can produce these kinds of test strings is:

https://www.levenshtein.net/fancy-text-generator

For transparency, I am involved with Levenshtein.net. I mention this tool because it generates copyable Unicode-styled text that can be used as test input; it is not being presented as an offline Notepad++ normalizer or a replacement for the PythonScript solution above.

The useful experiment is to classify which generated styles are covered by standard compatibility normalization and which require additional transformations.

For formal correctness testing, the Unicode standard and its conformance data should remain the authoritative references.

6. Why this also matters for text searching and fuzzy matching

Dean mentioned an important practical consequence: text that appears normal to a human reader may not match ordinary search queries.

This issue extends beyond Notepad++.

It also affects:

• Search indexes.
• Duplicate detection.
• Usernames and social media profiles.
• Fuzzy string matching.
• Text processing pipelines.
• Automated document comparison.

Consider:

Hello

𝗛𝗲𝗹𝗹𝗼

When compared as Unicode code-point sequences, a standard unit-cost Levenshtein calculation returns a distance of 5.

After NFKC normalization, the distance becomes 0.

The underlying edit-distance algorithm has not changed. The representation of the input text has changed.

This demonstrates why a text-matching system should explicitly define its normalization policy before calculating character-level similarity.

For a more detailed discussion of the relationship between normalization, transliteration, Unicode representation, and edit-distance algorithms, this technical explanation may be useful:

https://www.levenshtein.net/text-normalization-and-transliteration

Of course, normalization is not always desirable. A mathematical expression, a product identifier, or a username may intentionally distinguish characters that NFKC maps together.

For such applications, preserving the original string and maintaining a separate normalized search representation may be the safest approach.

7. A useful long-term improvement for Notepad++

If this functionality were considered for a future plugin or editor command, I think a small set of explicit options would be preferable to a single vaguely defined “convert to normal text” action:

• Canonical normalization (NFC).
• Compatibility normalization (NFKC).
• Optional diacritic removal.
• Optional application-specific styled-text cleanup.
• Selection-only or whole-document scope.
• Preview before replacement.
• Undo support.

The default should preserve the original text unless the user explicitly requests a transformation.

It would also be valuable to show how many characters or sequences will change before applying the operation.

Final thought

What I find particularly valuable about this discussion is that it identifies a practical problem created by the difference between visual typography and Unicode character identity.

The working PythonScript solution addresses many common styled-letter cases, while the remaining limitations highlight why Unicode normalization, transliteration, visual similarity, and encoding conversion must remain distinct concepts.

Thank you again to Mark Olson for sharing a practical implementation and to everyone who investigated the regex and Unicode behavior in detail.

I hope these additional considerations help users build safer and more predictable text-normalization workflows in Notepad++.

[Coises]

@Screen-White said:

plugin

For what it’s worth, I have a plugin called Unicode Normalize that can normalize selected text to any of the four standard Unicode normalization forms. There’s a short discussion of it here.

This plugin uses Windows’ NormalizeString function. It could probably be improved by using ICU4C. I haven’t done a lot of work on it; it’s just something I threw together to solve a problem.

The search problem is one I hope to solve someday in my work-in-progress called Search++, but it will be a while before I get to that.