Removing \u2018 and \u2019 character

Question

I am using Beautiful Soup to parse webpages and printing the name of the webpages visited on the terminal. However, often the name of the webpage has single left (\u2018) and right (\u2019) character which Python can't print as it gives a charmap encoding error.

Is there any way to remove these characters?

please add some of your code and your Output and expected output. it will be good to Draw better solutions — sundar nataraj, Jun 23 '14 at 04:46
Note: U+2018 = Left Single Quotation Mark, U+2019 = Right Single Quotation Mark — Marjorie Roswell, Jun 19 '17 at 05:27
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="UTF-8") is helpful to avoid encoding error in the first place. — Yiğit, Apr 23 '20 at 21:34

Martin Konecny · Accepted Answer · 2014-06-23T04:59:12.443

These codes are Unicode for the single left and right quote characters. You can replace them with their ASCII equivalent which Python shouldn't have any problem printing on your system:

>>> print u"\u2018Hi\u2019"
‘Hi’
>>> print u"\u2018Hi\u2019".replace(u"\u2018", "'").replace(u"\u2019", "'")
'Hi'

Alternatively with regex:

import re
s = u"\u2018Hi\u2019"
>>> print re.sub(u"(\u2018|\u2019)", "'", s)
'Hi'

However Python shouldn't have any problem printing the Unicode version of these as well. It's possible that you are using str() somewhere which will try to convert your unicode to ascii and throw your exception.

score 0 · Answer 2 · answered Jun 15 '23 at 22:16

Since single characters (\u2018 and \u2019) need to be replaced, str.translate() could be used as well.

s = "\u2018Hi\u2019"
s_new = s.translate(str.maketrans({'\u2018': "'", '\u2019': "'"}))

# or even use the unicode characters themselves
s_new = s.translate(str.maketrans({'‘': "'", '’': "'"}))
print(s_new)   # 'Hi'

Or specify the encoding in the first place when parsing.

# if parsing an html
soup = BeautifulSoup(html, from_encoding='utf-8')

# if reading a file
with open("text.txt", encoding='utf-8') as f:
     f.read()

Removing \u2018 and \u2019 character

2 Answers2

Linked