Read Unicode from CSV

Question

I have a problem reading unicode characters from a csv. The csv file originally had elements with unicode tags:

1. "[u'Aeron\xe1utica']"
2. "[u'Ni\u0161']"
3. "[u'K\xfcnste']" ...

from which I had to remove the u'' tags to give a csv with

1. Aeron\xe1utica
2. Ni\u0161
3. K\xfcnste ....

Now I want to read the csv and output it into a file with the characters i.e.

1. Aeronáutica
2. Niš
3. Künste ....

I tried using the UnicodeWriter in the csv docs, but it gives the same output as the second list

Here's what I did to read and write:

c = open('foo.csv','r')
r = csv.reader(c)
for row in reader:
p = p + row
#The elements in p were ['Aeron\\xe1utica', 'Ni\\u0161', 'K\\xfcnste'...]
c = open('bar.csv','w')
c.write(codecs.BOM_UTF8)
writer = UnicodeWriter(c)
for row in p:
writer.writerow([row])

I also tried codecs.open('','','UTF-8') for both reading and writing, but it didn't help

Answer 1

It appears you have written Python lists directly to your CSV file, resulting in the [...] literal syntax instead of normal columns. You then removed most of the information that could have been used to turn the information back to Python lists with unicode strings again.

What you have left are Python unicode literals, but without the quotes. Use the unicode_escape to decode the values to Unicode again:

with open('foo.csv','r') as b0rken
    for line in b0rken:
        value = line.rstrip('\r\n').decode('unicode_escape')
        print value

or add back the u'..' quoting, using a triple-quoted string in an attempt to avoid needing to escape embedded quotes:

with open('foo.csv','r') as b0rken
    for line in b0rken:
        value = literal_eval("u'''{}'''".format(line.rstrip('\r\n')))
        print value

If you still have the original file (with the [u'...'] formatted lines), use the ast.literal_eval() function to turn those back into Python lists. No point in using the CSV module here:

from ast import literal_eval

with open('foo.csv','r') as b0rken
    for line in b0rken:
        lis = literal_eval(line)
        value = lis[0]
        print value

Demo with unicode_escape:

>>> for line in b0rken:
...     print line.rstrip('\r\n').decode('unicode_escape')
... 
Aeronáutica
Niš
Künste
École de l'Air