How to move a python 2.6 project to UTF-8?

Question

We are moving from latin1 to UTF-8 and have 100k lines of python code.

Plus I'm new in python (ha-ha-ha!).

I already know that str() function fails when receiving Unicode so we should use unicode() instead of it with almost the same effect.

What are the other "dangerous" places of code?

Are there any basic guidelines/algorithms for moving to UTF-8? Can it be written an automatic 'code transformer'?

I've meant that the project has been using ASCII everywhere, and now the project should receive intentationalisation and should use Unicode everywhere — Dan, Mar 18 '11 at 12:22
Unicode strings don't use UTF-8 in Python (which doesn't limit their usefulness in any way), so please specify whether you want Unicode support *or* have to switch to UTF-8. For basic Unicode support in Python, you should prefer Python's Unicode strings (which use UTF-16 or UTF-32). — Philipp, Mar 19 '11 at 02:45
I wand Unicode support. Thanks for the notice with utf-8/utf-16 — Dan, Mar 21 '11 at 21:23

score 4 · Accepted Answer · answered Mar 18 '11 at 11:37

str and unicode are classes, not functions. When you call str(u'abcd') you are initialising a new string which takes 'abcd' as a variable. It just so happens that str() can be used to convert a string of any type to an ascii str.

Other areas to look out for are when reading from a file/input, or basically anything you get back as a string from a function that was not written for unicode.

Enjoy :)

score 2 · Answer 2 · answered Mar 18 '11 at 20:12

2

Can it be written an automatic 'code transformer'? =)

No. str and unicode are two different types which have different purposes. You should not attempt to replace every occurrence of a byte string with a Unicode string, neither in Python 2 nor Python 3.

Continue to use byte strings for binary data. In particular anything you're writing to a file or network socket is bytes. And use Unicode strings for user-facing text.

In between there is a grey area of internal ASCII-character strings which could equally be bytes or Unicode. In Python 2 these are typically bytes, in Python 3 typically Unicode. In you are happy to limit your code to Python 2.6+, you can mark your definitely-bytes strings as b'' and bytes, your definitely-characters strings as u'' and unicode, and use '' and str for the “whatever the default type of string is” strings.

answered Mar 18 '11 at 20:12

bobince

528,062
107
651
834

Files shouldn't necessarily be ascii, I would say that as soon as you find yourself using a character not in the first 127, it's better to use unicode, then you're not reliant on one codepage. – theheadofabroom Mar 18 '11 at 21:18
1

Files are explicitly bytes. You can't write characters directly to files. There are various methods for semi-automatically encoding Unicode characters sent to files (especially in Python 3), but after all explicit is better than implicit... – bobince Mar 19 '11 at 00:29
1

I think it's explicit enough if you state that you open a text file (using a `rt` or `wt` file mode). In Python 2.6 you have to use the `codecs` module, but that's still simpler than manual decoding and definitely explicit enough. – Philipp Mar 19 '11 at 02:47

score 0 · Answer 3 · edited May 23 '17 at 11:51

One way to quickly convert Python 2.x to have a default encoding of UTF-8 is to set the default encoding. This approach has its downsides--primarily that it changes the encoding for all libraries as well as your application, so use with caution. My company uses that technique in our production apps and it suits us well. It's also forward-compatible with Python 3, which has UTF-8 as the default encoding. You'll still have to change references of str() to unicode(), but you won't have to explicitly specify the encoding with .decode() and encode().

How to move a python 2.6 project to UTF-8?

3 Answers3