Removing hexadecimal characters from a unicode object

Question

I am trying to remove the hexadecimal characters \xef\xbb\xbf from my string however I am getting the following error.

Not quite sure how to resolve this.

>>> x = u'\xef\xbb\xbfHello'
>>> x
u'\xef\xbb\xbfHello'
>>> type(x)
<type 'unicode'>
>>> print x
ï»¿Hello
>>> print x.replace('\xef\xbb\xbf', '')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
>>>

score 3 · Accepted Answer · answered Nov 09 '16 at 12:29

You need to replace the unicode object, otherwise Python2 will to attempt to encode x with the ascii codec to search for the a str in it.

>>> x = u'\xef\xbb\xbfHello'
>>> x
u'\xef\xbb\xbfHello'
>>> print(x.replace(u'\xef\xbb\xbf',u''))
Hello

This only holds for Python2. In Python3 both versions will work.

score 0 · Answer 2 · edited May 23 '17 at 12:13

0

Try to use either the decode or unicode functions, like so:

x.decode('utf-8')

or

unicode(string, 'utf-8')

Source: UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 1

edited May 23 '17 at 12:13

Community

1
1

answered Nov 09 '16 at 12:34

Henrique Moisés

102
9

score 0 · Answer 3 · answered Nov 09 '16 at 16:32

The real problem was that your Unicode string was incorrectly decoded in the first place. Those characters are a UTF-8 byte order mark (BOM) character mis-decoded as (likely) latin-1 or cp1252.

Ideally, fix how they were decoded, but you can reverse the error by re-encoding as latin1 and decoding correctly:

>>> x = u'\xef\xbb\xbfHello'
>>> x.encode('latin1').decode('utf8') # decode correctly, U+FEFF is a BOM.
u'\ufeffHello'
>>> x.encode('latin1').decode('utf-8-sig') # decode and handle BOM.
u'Hello'

Removing hexadecimal characters from a unicode object

3 Answers3