Handling escaped unicode characters from JSON response

Question

I'm receiving a list of location names like this

region_response = json.loads(requests.get(region_req_url).text)

Many of the names have characters represented like Tor\u00f6 Stenstrand in the response, where that place name would be Torö Stenstrand.

I'm then adding some of the received elements to a dictionary and saving that to a new JSON file

with open('spots.json', 'w') as wf:
    json.dump(results, wf, skipkeys=False, ensure_ascii=True, indent=4)

The resulting file also has the escaped characters like \u00f6 but I need this to have the actual representations like ö.

My work so far is in this repo, specifically in magicseaweed.py and windguru.py.

Apologies if this has been answered before, or if my description/assumptions above are incorrect - I've been trying to work this out for quite a while now, but I don't think I understand the area enough to know exactly what I should be looking for or reading up on! Any help/suggestions/pointers would be massively appreciated.

score 1 · Answer 1 · answered May 26 '17 at 12:26

1

I once solve this issue with io module as follows

import io
j = {'d': '中', 'e': 'a'}
with io.open('myfile.json', 'w', encoding='utf8') as json_file:
   json.dumps(j, json_file, ensure_ascii=False)

output:

    {"d": "中", "e": "a"}'

answered May 26 '17 at 12:26

Sijan Bhandari

2,941
3
23
36

Thank you for your answer! I'm getting an error because my dictionary contains strings not unicode `TypeError: must be unicode, not str` - so the escaped characters like `\u00f6` are in string format and will need to be converted to unicode. I've tried `unidict = {k.decode('utf8'): v.decode('utf8') for k, v in results.items()}` from [this answer](https://stackoverflow.com/questions/16705274/fastest-way-to-convert-a-dicts-keys-values-from-str-to-unicode) but getting error `AttributeError: 'dict' object has no attribute 'decode'`, so I think I'm missing something here. – Cian May 27 '17 at 00:46

score 0 · Accepted Answer · answered May 29 '17 at 11:06

Got this working :) First I added encoding to the json.loads...

response = requests.get(region_req_url)
response = requests.utils.get_unicode_from_response(response)
region_response = json.loads(response, encoding='utf-8')

Then after trying Sijan's answer above I was getting errors (shown in comment), and when trying to fix the error I came across this answer, and that solution has worked. So to write to my JSON file I'm doing...

with io.open("results.json",'w', encoding="utf-8") as outfile:
    outfile.write(unicode(json.dumps(results, ensure_ascii=False, indent=4)))

Now place names are writing to the JSON file like Torö Stenstrand.

Handling escaped unicode characters from JSON response

2 Answers2