Why did I get UnicodeDecodeError when I read a file which contains Chinese characters?

Question

>>> path = 'name.txt'
>>> content = None
>>> with open(path, 'r') as file:
...     content = file.readlines()
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/mnt/lustre/share/miniconda3/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 163: ordinal not in range(128)

When I run this code to read a file which contains Chinese characters, I got an error. The file is saved by using UTF-8. My python version is 3.6.5. But it runs ok in python2.7.

open can take an optional "encoding" keyword in python3 ... in python 2 it would implicitly try to decode it for you ... most of the time it got it right ... — Joran Beasley, Feb 12 '20 at 06:04
Did you read the error message? `'ascii' codec` ... You need to tell it to use UTF-8. — wjandrea, Feb 12 '20 at 06:06
@sinafarhadi No, that would open it as binary. OP wants to read UTF-8. — wjandrea, Feb 12 '20 at 06:06
@Joran Python 2's `open()` doesn't decode. It just returns byte strings. You may be thinking of `codecs.open()` or `io.open()`. — Mark Tolonen, Feb 12 '20 at 06:28
@sinafarhadi `rb` can open it without errors, but it unreadable. — Hu Zhang, Feb 12 '20 at 08:19

score 1 · Accepted Answer · answered Feb 12 '20 at 06:18

1

open is using the ASCII codec to try to read the file. The easiest way to fix this is to specify the encoding:

with open(path, 'r', encoding='utf-8') as file:

Your locale should probably specify the preferred encoding as UTF-8, but I think it depends on OS and language settings.

answered Feb 12 '20 at 06:18

wjandrea

28,235
9
60
81

I can read the file without errors by using this. But when I try `print (content[0])`, the same error occurs. – Hu Zhang Feb 12 '20 at 08:10

score 1 · Answer 2 · answered Feb 12 '20 at 06:21

Python 2.7 reads files into byte strings by default.

Python 3.x reads files into Unicode strings by default, so the bytes in the file must be decoded.

The default encoding used varies by operating system, but can be determined by calling locale.getpreferredencoding(False). This is often utf8 on Linux systems, but Windows systems return a localized ANSI encoding, e.g. cp1252 for US/Western European Windows versions.

In Python 3, specify the encoding you expect for files so as not to rely on a locale-specific default. For example:

with open(path,'r',encoding='utf8') as f:
    ...

You can do this in Python 2 as well, but use io.open(), which is compatible with Python 3's open() and will read Unicode strings instead of byte strings. io.open() is available in Python 3 as well for portability.

Why did I get UnicodeDecodeError when I read a file which contains Chinese characters?

2 Answers2