Python 2.7 unicode confusion again

Question

I've already read this:

Setting the correct encoding when piping stdout in Python

And I'm trying to stick with the rule of thumb: "Always use Unicode internally. Decode what you receive, and encode what you send."

So here's my main file:

# coding: utf-8

import os
import sys

from myplugin import MyPlugin
if __name__ == '__main__':
    c = MyPlugin()
    a = unicode(open('myfile.txt').read().decode('utf8'))
    print(c.generate(a).encode('utf8'))

What is getting on my nerves is that:

• I read in a utf8 file so I decode it.
• then I force convert it to unicode which gives unicode(open('myfile.txt').read().decode('utf8'))
• then I try to output it to a terminal
• on my Linux shell I need to re-encode it to utf8, and I guess this is normal because I'm working all this time on an unicode string, then to output it, I have to re-encode it in utf8 (correct me if I'm wrong here)
• when I run it with Pycharm under Windows, it's twice utf8 encoded, which gives me things like agréable, déjÃ. So if I remove encode('utf8') (which changes the last line to print(c.generate(a)) then it works with Pycharm, but doesn't work anymore with Linux, where I get: 'ascii' codec can't encode character u'\xe9' in position blabla you know the problem.

If I try in the command line:

• Linux/shell ssh: import sys sys.stdout.encoding I get 'UTF-8'
• Linux/shell in my code: import sys sys.stdout.encoding I get None WTF??
• Windows/Pycharm: import sys sys.stdout.encoding I get 'windows-1252'

What is the best way to code this so it works on both environments?

Answer 1

unicode(open('myfile.txt').read().decode('utf8'))

no need to wrap with unicode because result of str.decode is already unicode.

print(c.generate(a).encode('utf8'))

no need to encode because Python will encode the string itself depending on the terminal encoding.

So this is the correct way to do

print(c.generate(a))

You are getting 'ascii' codec can't encode character u'\xe9' in position because your Linux terminal has ascii encoding, so it's not possible for Python to print unicode characters to it.

See https://wiki.python.org/moin/PrintFails

I would suggest fixing your terminal (environment), not the code. You should not depend on the terminal encoding, especially as usually you print this info to a file.

If you still want to print it to any terminal which supports ASCII, you can use str.encode('unicode-escape'):

>>> print(u'щхжы'.encode('unicode-escape'))
\u0449\u0445\u0436\u044b

But it will be not very readable by humans, so I don't see the point.

Answer 2

You're philosophy is correct but you're over complicating things and making your code brittle.

Open files in text mode to automatically convert to Unicode for you. Then print without encoding - print is supposed to work out the correct encoding.

If your Linux environment isn't set correctly, then set PYTHONIOENCODING=utf-8 in your Linux environment vars (export PYTHONIOENCODING=utf-8) to fix up any issues during print. You should consider setting your locale to a UTF-8 variation such as en_GB.UTF-8 to avoid having to define the PYTHONIOENCODING.

PyCharm should work without modification.

Your code should look like:

import os
import sys
import io

from myplugin import MyPlugin

if __name__ == '__main__':
    c = MyPlugin()
    # t is the default
    with io.open('myfile.txt', 'rt', encoding='utf-8') as myfile:
        # a is now a Unicode string
        a = myfile.read()

    result = c.generate(a)
    print result

If you're using Python 3.x, drop import io and io. from io.open().