0

I'm having trouble resolving an encoding error when reading a csv file using the pandas library.

import pandas as pd
filepath = "D:\Datasets\2019HighwayBridgeInventory"
pd.read_csv(filepath + '\2019HwyBridgesDelimitedUtah.csv')

This returns a UnicodeDecodeError:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 11: invalid start byte

I understand this error occurs when a non UTF8 character exists in the data after reviewing this relevant thread. However, my attempts to resolve the error have been unsuccessful.

At first I tried opening the file and saving it with utf8 encoding in sublime text, but received the same error message.

I have also tried specifying the encoding in the read_csv statement. I tried

pd.read_csv(filepath + '\2019HwyBridgesDelimitedUtah.csv', encoding = "ISO-8859–1")

pd.read_csv(filepath + '\2019HwyBridgesDelimitedUtah.csv', encoding = "us-ascii")

pd.read_csv(filepath + '\2019HwyBridgesDelimitedUtah.csv', encoding = "latin1")

but I seem to receive the same utf encoding error every time. is it possible that this error is not related to the read_csv statement? Why is the error still stating the utf-8 codec cant read something even when I change the encoding to something else?

Full error text:

UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-1e8cb6445435> in <module>
----> 1 pd.read_csv(filepath + '\2019HwyBridgesDelimitedUtah.csv', encoding = "iso-8859-1")

~\Anaconda3\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
    700                     skip_blank_lines=skip_blank_lines)
    701 
--> 702         return _read(filepath_or_buffer, kwds)
    703 
    704     parser_f.__name__ = name

~\Anaconda3\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
    427 
    428     # Create the parser.
--> 429     parser = TextFileReader(filepath_or_buffer, **kwds)
    430 
    431     if chunksize or iterator:

~\Anaconda3\lib\site-packages\pandas\io\parsers.py in __init__(self, f, engine, **kwds)
    893             self.options['has_index_names'] = kwds['has_index_names']
    894 
--> 895         self._make_engine(self.engine)
    896 
    897     def close(self):

~\Anaconda3\lib\site-packages\pandas\io\parsers.py in _make_engine(self, engine)
   1120     def _make_engine(self, engine='c'):
   1121         if engine == 'c':
-> 1122             self._engine = CParserWrapper(self.f, **self.options)
   1123         else:
   1124             if engine == 'python':

~\Anaconda3\lib\site-packages\pandas\io\parsers.py in __init__(self, src, **kwds)
   1851         kwds['usecols'] = self.usecols
   1852 
-> 1853         self._reader = parsers.TextReader(src, **kwds)
   1854         self.unnamed_cols = self._reader.unnamed_cols
   1855 

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()

~\Anaconda3\lib\genericpath.py in exists(path)
     17     """Test whether a path exists.  Returns False for broken symbolic links"""
     18     try:
---> 19         os.stat(path)
     20     except OSError:
     21         return False

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 11: invalid start byte
Sean Diller
  • 1
  • Can you please try: `data = pd.read_csv(filename, encoding= 'unicode_escape')` – Kuldeep Singh Sidhu Dec 06 '20 at 04:55
  • I suggest you try to load the file with Notepad++. It'll show you (bottom status bar) the actual encoding, and you won't have to guess. 2nd, I suggest you'll look at the start of the file to make sure there really aren't any unexpected characters. Finally, I suggest you get rid of the '\' slashes in your path and replace them with '/' since '\' is also an escape character in python. Alternatively, you may try to use a preceding r, like so: r"D:\Datasets\2019HighwayBridgeInventory". Otherwise python will see this string as if you're trying to escape the 'D' character in 'Datasets' – Ruslan Dec 06 '20 at 06:17

2 Answers2

0

A suggestion would be to check which encoding you actually have. Do it this way:

with open('filename.csv) as f:  ### or whatever your extension is
   print(f)

from that you'll obtain the encoding. Then,

df=pd.read_csv('filename.csv', encoding="the encoding that was returned")
Serge de Gosson de Varennes
  • 7,162
  • 3
  • 18
  • 39
  • ??? `with open('anyfile) as f: print(f)` _always_ returns `encoding='cp65001'` even for files with `UTF16LEBOM` or `UTF16BEBOM`. Windows 10 64 bit, Python 3.8.6… – JosefZ Feb 20 '21 at 14:46
0

If you use python3.0 or more higher, you can code like me, for example:

df = pd.read_csv(f'E:\\每月笔记采集数据文件\\11月笔记采集任务\\{name}', encoding='ISO-8859-1')

if you have done it, you will find the decode type, ISO-8859-1, can decode almost all types. Thanks for your reading.

biao zhu
  • 11
  • 3