UTF-8 encoding issues

Question

I'm upgrading a Rails app from 2 to 3. I have a Mysql table with data like this:

½†x 12†Rebar          
14’ LVL 1-3/4" X 11-7/8"  

When running Rails 2 these names look normal (½” x 12” Rebar and 14’ LVL 1-3/4" X 11-7/8"), but when I run my Rails 3 branch Ruby is using the same form as the database.

My guess is Rails 2 ignores the table encoding for another instead. Somehow I need to force the encoding, not convert it. I fixed this issue for one string through use of <string>.encode().force_encoding() but I'd rather fix the root cause in the data.

Answer 1

In the end I identified all the characters that were breaking down and replaced them with Find-Replaces and then reimported the data into a utf-8 table. On a whim I tried to use this data in the live environment and the live environment display broke in a similar way even though the modified data looked great in development.

Diffing the old data and the new data reveals how messed up the data is:

½â€<U+009D> x 12â€<U+009D> Rebar

becomes

½\" x 12\" Rebar

I'm guessing the encoding Rails 2 is using isn't UTF-8.