Error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Question

https://github.com/affinelayer/pix2pix-tensorflow/tree/master/tools

An error occurred when compiling "process.py" on the above site.

python tools/process.py --input_dir data --operation resize --output_dir data2/resize
data/0.jpg -> data2/resize/0.png

Traceback (most recent call last):
  File "tools/process.py", line 235, in <module>
    main()
  File "tools/process.py", line 167, in main
    src = load(src_path)
  File "tools/process.py", line 113, in load
    contents = open(path).read()
  File"/home/user/anaconda3/envs/tensorflow_2/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

What is the cause of the error? Python's version is 3.5.2.

Alfe · Accepted Answer · 2018-11-20T09:21:26.927

374

Python tries to convert a byte-array (a bytes which it assumes to be a utf-8-encoded string) to a unicode string (str). This process of course is a decoding according to utf-8 rules. When it tries this, it encounters a byte sequence which is not allowed in utf-8-encoded strings (namely this 0xff at position 0).

Since you did not provide any code we could look at, we only could guess on the rest.

From the stack trace we can assume that the triggering action was the reading from a file (contents = open(path).read()). I propose to recode this in a fashion like this:

with open(path, 'rb') as f:
  contents = f.read()

That b in the mode specifier in the open() states that the file shall be treated as binary, so contents will remain a bytes. No decoding attempt will happen this way.

edited Nov 20 '18 at 09:21

answered Feb 20 '17 at 09:26

Alfe

56,346
20
107
159

I'm getting the error "ValueError: mode string must begin with one of 'r', 'w', 'a' or 'U', not 'br'" – Unnikrishnan Apr 03 '17 at 10:10
5

@Unnikrishnan Ok, then use `rb` (I thought order was of no importance, but it seems to be, at least in some systems/versions). I changed my answer accordingly. – Alfe Apr 03 '17 at 11:54
127

`byte 0xff in position 0` could also mean the file is encoded in UTF-16, then you can do `with open(path, encoding='utf-16') as f:` instead – Nikolai R Kristiansen Sep 27 '17 at 11:12
What if there is actually no `0xff` character at position `0`? And it is `UTF-8` encoded. – Iulian Onofrei Sep 02 '19 at 11:26
A pure `'\xFF'` character will be encoded in UTF-8 as `'\xC3\xBF'`. UTF-8 encodes all characters with a set MSB using two characters. (See the output of `printf "\xff" | iconv -f latin1 -t utf-8 | xxd` in a shell.) A verbatim `'\xFF'` in the beginning of a UTF-8 encoded string is an encoding error (could be called a syntax error in terms of UTF-8). – Alfe Sep 02 '19 at 15:17
@NikolaiRKristiansen it's not. b'\xff'.decode('utf16') => UnicodeDecodeError: 'utf-16-le' codec can't decode byte 0xff in position 0: truncated data – Zim Jan 24 '20 at 19:42
2

I think this is the correct answer: when dealing with binary files the encoding is not involved, and should not be, at all. – lorenzo Mar 31 '20 at 09:58
1

@Zim The `\xff` in the beginning is not _part_ of the file contents but part of a _marker_ which tells us the encoding _of the rest_ of the file. So read binary (without decoding): `b = open(..., 'rb').read()` and then check for the encoding and decode: `if b[0] == b'\xff': return b[1:].decode('utf-16')`. IIRC the marker for utf-16 is `\xff` and a second byte, not just one byte. – Alfe Feb 24 '21 at 09:53

score 163 · Answer 2 · answered Feb 01 '18 at 05:53

163

Use this solution it will strip out (ignore) the characters and return the string without them. Only use this if your need is to strip them not convert them.

with open(path, encoding="utf8", errors='ignore') as f:

Using errors='ignore' You'll just lose some characters. but if your don't care about them as they seem to be extra characters originating from a the bad formatting and programming of the clients connecting to my socket server. Then its a easy direct solution. reference

answered Feb 01 '18 at 05:53

Nitish Kumar Pal

2,738
3
18
23

13

Works for decode() as well: `contents = contents.decode('utf-8', 'ignore')` Source: https://docs.python.org/3/howto/unicode.html#the-string-type – naaman Jun 09 '18 at 09:45
1

When you say "lose some characters" do you mean that the file with errors won't be read? or that not all the content of that file will be read? – msoutopico Dec 10 '19 at 15:57
@msoutopico As it is ignoring the errors, so some encodings won't be read which are causing issues. But haven't ever come across any content that has been skipped while reading. So basically ecoding issues are ignored. – Nitish Kumar Pal Dec 12 '19 at 09:35
@NitishKumarPal, ok so no real content should be skipped or lost – msoutopico Dec 12 '19 at 17:55
I dont understand, it did the work but I didnt skipped any csv row. Not sure what happened here. – im_bhatman May 06 '20 at 17:46
Resolved the issue of converting HEX into CHAR representation using this encoding. Thanks a lot – JEEVA RAJA PAUL Aug 23 '21 at 08:41
thanks I kept getting errors reading a sql file using utf-8 on some of the files. When I added ignore errors I was able to continue processing the sqls. – Golden Lion Nov 09 '22 at 22:58

score 78 · Answer 3 · answered Jun 04 '19 at 20:04

78

Use encoding format ISO-8859-1 to solve the issue.

answered Jun 04 '19 at 20:04

Ramineni Ravi Teja

3,568
26
37

4

This will hide the error but produce garbage if the *actual* encoding is not ISO-8859-1. If you are not sure, examine some of the strings with character codes in the range 128-255. Maybe see also https://tripleee.github.io/8bit/ – tripleee Aug 04 '20 at 16:45
2

This will eliminate errors, but only because ISO-8859-1 defines a character for *each* one of the 256 possible byte values. They won't necessarily be the *right* characters and you need to verify that you're reading the correct text despite the lack of errors. – Mark Ransom Nov 09 '20 at 04:22
1

Some times it will give correct characters but most of times it will give garbage values like "1.5 1 0 obj > endobj 2 0 obj > endobj 4 0 obj > stream x½½þù/qyúßù§ÿ¢ÿèÿþðçõ¯ÿø¿þòÿG\ü;x¯¯oüùïó_÷þýòÿøß~ù¿ùå/þ¡îÝR^?/jáòòòüþô~ÿ|þx}L_¿}^__.÷ÛóçûÓëççóíöôöúòüÒWÿú¿x¿0´ÍIâ èÛå)ä¼{$éúÎ oÎçåùóZØil¬Pÿá$0JÏ{²úñsr^n – Yashraj Nigam May 04 '21 at 11:13
No, it only has 256, but it means every byte corresponds to one character, and thus you will not ever get any errors. If this wasn't the correct encoding, you get [mojibake](https://en.wikipedia.org/wiki/Mojibake) in your output instead. – tripleee Nov 05 '22 at 19:38
@YashrajNigam You are overgeneralizing. It sounds like most of the time you are processing data which uses a completely different encoding, but that is untrue for many visitors here. The real solution, as several of the comments and answers here try to explain, is to establish the correct character encoding in each case. – tripleee Nov 05 '22 at 19:41
This worked for me. I'm probably doing this way wrong,but in Google Colab I was saving to my GoogleDrive as a doc with the extension `.docx`. Even though i was specifiying "UTF-8" when writing the file. I guess I still had characters that were not UTF-8 because reading the file in the same format would generate errors. But when reading with the format above, the file rendered perfectly – enjoi4life411 Jan 30 '23 at 01:48

score 36 · Answer 4 · answered Aug 16 '17 at 15:34

36

Had an issue similar to this, Ended up using UTF-16 to decode. my code is below.

with open(path_to_file,'rb') as f:
    contents = f.read()
contents = contents.rstrip("\n").decode("utf-16")
contents = contents.split("\r\n")

this would take the file contents as an import, but it would return the code in UTF format. from there it would be decoded and seperated by lines.

answered Aug 16 '17 at 15:34

tattmoney76

492
4
6

14

In Python 3 you can simplify this by using the encoding param `with open(path, encoding='utf-16') as f` – Nikolai R Kristiansen Sep 27 '17 at 11:14
@NikolaiRKristiansen I tried using your method, but got an error as ```TypeError: an integer is required (got type str)```. Why? Both files are binary and read as ```rb```. – Bogota Jun 17 '20 at 17:20
2

@Bogota The `encoding` param only makes sense when reading text. Drop the 'b' from the mode argument and try again. Read more in the docs: https://docs.python.org/3/library/functions.html#open – Nikolai R Kristiansen Jun 17 '20 at 21:14

Peter Ogden · Answer 5 · 2017-12-05T08:37:19.620

25

I've come across this thread when suffering the same error, after doing some research I can confirm, this is an error that happens when you try to decode a UTF-16 file with UTF-8.

With UTF-16 the first characther (2 bytes in UTF-16) is a Byte Order Mark (BOM), which is used as a decoding hint and doesn't appear as a character in the decoded string. This means the first byte will be either FE or FF and the second, the other.

Heavily edited after I found out the real answer

edited Dec 05 '17 at 08:37

answered Dec 04 '17 at 13:01

Peter Ogden

819
8
5

1

This ended 2 hours of headache! Opening the file with open('filename', 'r') as f: and then printing its contents shows UTF-8, which is wrong. – nulldroid Dec 14 '19 at 14:21
This one worked for my case. `with open(filename, encoding='utf-16') as f:` – nathanielng Apr 16 '23 at 02:09

score 11 · Answer 6 · edited Aug 23 '21 at 09:34

11

Those getting similar errors while handling Pandas for data frames use the following solution.

example solution.

df = pd.read_csv("File path", encoding='cp1252')

edited Aug 23 '21 at 09:34

lejlun

4,140
2
15
31

answered Aug 23 '21 at 08:06

13Tracso

479
4
8

2

This is where I ended up, without knowing this answer. Just checked in this thread whether someone answered like this, and yes - someone did. – questionto42 Sep 30 '21 at 14:03
1

Randomly guessing at a different encoding might remove the error, but could then produce garbage results. The useful answer would be a way to figure out which encoding is actually correct. – tripleee Nov 05 '22 at 19:31
1

You might be right, with that, you can use chardet to auto-detect the file type encoding. Maybe I should also state that this solution directly applies to reading CSV files created from Microsoft excel. – 13Tracso Nov 21 '22 at 11:25

score 9 · Answer 7 · answered Nov 24 '20 at 11:22

9

I had a similar issue with PNG files. and I tried the solutions above without success. this one worked for me in python 3.8

with open(path, "rb") as f:

answered Nov 24 '20 at 11:22

Nwawel A Iroume

1,249
3
21
42

score 8 · Answer 8 · answered Nov 10 '19 at 19:32

8

It simply means that one chose the wrong encoding to read the file.

On Mac, use file -I file.txt to find the correct encoding. On Linux, use file -i file.txt.

answered Nov 10 '19 at 19:32

Minh Triet

1,190
1
15
35

`file` is very imprecise when it comes to guessing encodings. You can try the Python libraries `chardet` or `ftfy` but they too are heuristic tools, not infallible oracles. If you know or are able to guess what text to expect, you can look up problem bytes at https://tripleee.github.io/8bit/ and figure out at least a good guess for your data. – tripleee Nov 05 '22 at 19:36

score 7 · Answer 9 · edited Jun 17 '18 at 14:14

7

use only

base64.b64decode(a)

instead of

base64.b64decode(a).decode('utf-8')

edited Jun 17 '18 at 14:14

Paul Roub

36,322
27
84
93

answered Jun 17 '18 at 13:34

pradeep karunathilaka

578
6
6

4

its working but just to understand can you explian why please? :) – Ido Bleicher Mar 27 '19 at 12:58
especially, where do you use it? What is 'a'? – Vincenzooo Jul 11 '22 at 15:16
removing `decode` solved an issue I was having too. – uak Aug 25 '22 at 05:36

Jie Yin · Answer 10 · 2020-12-17T01:08:02.587

This is due to the different encoding method when read the file. In python, it defaultly encode the data with unicode. However, it may not works in various platforms.

I propose an encoding method which can help you solve this if 'utf-8' not works.

with open(path, newline='', encoding='cp1252') as csvfile:
    reader = csv.reader(csvfile)

It should works if you change the encoding method here. Also, you can find other encoding method here standard-encodings , if above doesn't work for you.

score 5 · Answer 11 · edited Dec 29 '21 at 00:13

I had this UnicodeDecodeError while trying to read a '.csv' file using pandas.read_csv(). In my case, I could not manage to overcome this issue using other encoder types. But instead of using

pd.read_csv(filename, delimiter=';')

I used:

pd.read_csv(open(filename, 'r'), delimiter=';')

which just seems working fine for me.

Note that: In open() function, use 'r' instead of 'rb'. Because 'rb' returns bytes object that causes to happen this decoder error in the first place, that is the same problem in the read_csv(). But 'r' returns str which is needed since our data is in .csv, and using the default encoding='utf-8' parameter, we can easily parse the data using read_csv() function.

Saif Faidi · Answer 12 · 2019-05-23T08:18:11.973

2

if you are receiving data from a serial port, make sure you are using the right baudrate (and the other configs ) : decoding using (utf-8) but the wrong config will generate the same error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

to check your serial port config on linux use : stty -F /dev/ttyUSBX -a

edited May 23 '19 at 08:18

answered Apr 19 '19 at 10:36

Saif Faidi

509
4
15

score 2 · Answer 13 · answered Jul 30 '20 at 23:55

I had a similar issue and searched all the internet for this problem

if you have this problem just copy your HTML code in a new HTML file and use the normal <meta charset="UTF-8"> and it will work....

just create a new HTML file in the same location and use a different name

score 1 · Answer 14 · edited Jul 04 '17 at 16:05

1

Check the path of the file to be read. My code kept on giving me errors until I changed the path name to present working directory. The error was:

newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

edited Jul 04 '17 at 16:05

luchonacho

6,759
4
35
52

answered Jul 04 '17 at 14:19

Rex131xO

11
2

score 1 · Answer 15 · answered Jan 21 '19 at 19:49

1

If you are on a mac check if you for a hidden file, .DS_Store. After removing the file my program worked.

answered Jan 21 '19 at 19:49

Juan

477
4
8

Are you sure you answered the correct question? What has a `.DS_Store` file to do with the Python `UnicodeDecodeError`? – wovano Nov 18 '22 at 22:05

score 0 · Answer 16 · answered Nov 01 '19 at 10:50

I had a similar problem.

Solved it by:

import io

with io.open(filename, 'r', encoding='utf-8') as fn:
  lines = fn.readlines()

However, I had another problem. Some html files (in my case) were not utf-8, so I received a similar error. When I excluded those html files, everything worked smoothly.

So, except from fixing the code, check also the files you are reading from, maybe there is an incompatibility there indeed.

score -1 · Answer 17 · answered Aug 15 '20 at 08:07

You have to use the encoding as latin1 to read this file as there are some special character in this file, use the below code snippet to read the file.

The problem here is the encoding type. When Python can't convert the data to be read, it gives an error.

You can you latin1 or other encoding values.

I say try and test to find the right one for your dataset.

score -2 · Answer 18 · answered May 19 '20 at 07:48

-2

I have the same issue when processing a file generated from Linux. It turns out it was related with files containing question marks..

answered May 19 '20 at 07:48

wfolkerts

99
1
4

Could you explain why Linux or question marks have anything to do with this question? The question mark can be encoded/decoded using ASCII, UTF-8, Latin-1 and many more encodings without problems, so I don't see how it could cause a `UnicodeDecodeError`. – wovano Nov 18 '22 at 22:08

score -3 · Answer 19 · answered Jun 18 '22 at 23:10

-3

Following code worked in my case:

df = pd.read_csv(filename,sep = '\t', encoding='cp1252')

answered Jun 18 '22 at 23:10

Gajanan Kothawade

35
6

1

This is just a repeat of an earlier answer. – tripleee Nov 05 '22 at 19:31

score -6 · Answer 20 · answered Aug 13 '17 at 13:48

-6

If possible, open the file in a text editor and try to change the encoding to UTF-8. Otherwise do it programatically at the OS level.

answered Aug 13 '17 at 13:48

Manoj Joshi

13
1

Error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

20 Answers20

Linked

Related