@Tom-Sasson,
DOWNLOAD_FOLDER = r"C:\Users\MYUSER\Documents\עבודה\SOMFOLDER\תלושים"
Unfortunately, that alone isn’t enough to replicate the problem for us. If I save that line of code into a UTF-8-encoded file, and open it in Notepad++, it is properly recognized and interpreted as UTF-8.
As @Coises suggested, if you could share enough code that we could replicate it, it would be great. (See below for a way to hexdump the file, to make sure we can create the same file as you, even if we don’t have pycharm available.)
If there is proprietary information in your code, could you pare it down to the minimum UTF-8 file that you can create in Pycharm that shows the same problem when loaded in Notepad++? If not, you will have to do more investigation on your side with us randomly guessing as to how to direct you: as @Coises suggested, my first steps would be to turn off the autodetect, as no encoding autodetect works 100% of the time; once that’s off, you might also want to see if changing Settings > Preferences > New Document to choose UTF-8 and ☑ Apply to opened ANSI files in the Encoding box on that page of settings.
Normally, with the symptoms you gave, I would guess that Notepad++ saw too many “invalid” UTF-8 sequences: UTF-8 has limited sets of high-order byte sequences that are allowed – for example, after a normal ASCII character, a high-order byte will never be less than 0xC0 – and a bunch more rules like that, where this table in the Wikipedia UTF-8 article gives the rules for valid UTF-8 high-byte sequences.
Whether you are providing us with a minimal file to replicate, or whether you are investigating proprietary files on your own, the next thing I would suggest is a way to hexdump the file – so you can see what bytes pycharm is writing to disk, vs what characters Notepad++ is interpreting those characters. Since you mentioned pycharm, I am assuming you have Python3 installed on your computer, so save the following as hexdump.py:
import sys;
data=open(sys.argv[1],'rb').read();
offset=0;
while offset < len(data):
chunk = data[offset : offset + 16];
hex_values = ' '.join(f'{byte:02x}' for byte in chunk);
ascii_values = ''.join(chr(byte) if 32 <= byte <= 126 else '.' for byte in chunk);
print(f'{offset:08x} {hex_values:<48} |{ascii_values}|');
offset += 16;
Then, if your file created by pycharm is called tom.py, run python hexdump.py tom.py
C:\usr\local\share\TempData\Npp>python hexdump.py tom.py
00000000 44 4f 57 4e 4c 4f 41 44 5f 46 4f 4c 44 45 52 20 |DOWNLOAD_FOLDER |
00000010 3d 20 72 22 43 3a 5c 55 73 65 72 73 5c 4d 59 55 |= r"C:\Users\MYU|
00000020 53 45 52 5c 44 6f 63 75 6d 65 6e 74 73 5c d7 a2 |SER\Documents\..|
00000030 d7 91 d7 95 d7 93 d7 94 5c 53 4f 4d 46 4f 4c 44 |........\SOMFOLD|
00000040 45 52 5c d7 aa d7 9c d7 95 d7 a9 d7 99 d7 9d 22 |ER\............"|
If this is the “minimal file” with no proprietary info, then paste that hexdump along with the raw text of the file when you reply.
If you are just investigating on your own, then look for the first character where you have a Hebrew character in pycharm but it shows up as Chinese (or whatever) in Notepad++: for example, the end of line 3 in the dump is 5c (the \ backslash), followed by d7 a2 (which is ע: mixed LTR and RTL is really confusing to me, so that surprised me at first). Anyway, you can look for “problematic” sequences, like anytime a small byte (00-7F) is followed by a “medium” byte (80-BF), which isn’t allowed; or anytime a high byte (C0-F7) is followed by anything other than a “medium byte”. Maybe you’ll find that pycharm is saving an invalid UTF-8 (though I doubt it).
