Write a string with non-ASCII characters in it - error only if string is a variable?

Question

I'm trying to write strings with non-ASCII characters in it to a file, such as "maçã", "pé", and so on.

I'm currently doing something like this:

_setmode(_fileno(stdout), _O_U16TEXT);

//I added the line above recently to the question,
//but it was in the code before, I forgot to write it
//I also included some header files, to be able to do that
//can't really remember which, if necessary I'll look it up.


wstring word=L"";
wstring file = L"example_file.txt"
vector<wstring> my_vector;

wofstream my_output(file);

while(word != L".")
{
 getline(wcin, word);
 if(word!= L".")
   my_vector.pushback(word);
}

for(std::vector<wstring>::iterator j=my_vector.begin(); j!=my_vector.end(); j++)
    {
        my_output << *j << endl;
//element pointed by iterator going through the whole vector

        my_output << L("maçã pé") << endl;
    }
my_output.close();

Now, if I enter "maçã", "pé" and "." as words (only the 1st two are stored in the vector), the output to the file is rather strange:

• the words I entered (stored in variables) appear strange: "ma‡Æ" and "p,";
• the words stored directly in the code appear perfectly normal "maçã pé";

I have tried using wcin >> word instead of getline(wcin, word) and writing to the console instead of a file, the results are the same: writes variable strings wrong, writes strings directly in code perfectly.

I cannot find a reason for this to happen, so any help will be greatly appreciated.

Edit: I am working in Windows 7, using Visual C++ 2010

Edit 2: added one more line of code, that I had missed. (right in the beginning)

EDIT 3: following SigTerm's suggestion, I realised the problem is with the input: neither wcin nor getline are getting the strings with right formatting to variable wstring word. So, the question is, do you know what is causing this or how to fix it?

Answer 1

Try to include

#include <locale>

and at the beginning of main, write

std::locale::global(std::locale(""));

Answer 2

Windows makes encodings confusing because the console typically uses an "OEM" code page, while GUI applications use an "ANSI" code page. The each vary with the localized version of Windows used. On U.S. Windows, The OEM code page is 437 and the ANSI code page is 1252.

Keeping the above in mind, setting the streams to the locale being used fixes the problem. If working in the console, use the console's code page:

wcin.imbue(std::locale("English_United States.437"));
wcout.imbue(std::locale("English_United States.437"));

But keep in mind most code pages are single-byte encodings, so only understand 256 possible Unicode characters:

wstring word;
wcin.imbue(std::locale("English_United States.437"));
wcout.imbue(std::locale("English_United States.437"));
getline(wcin, word);
wcout << word << endl;
wcout << L"maçã pé" << endl;

This returns on the console:

maça pé
maça pé

Code page 437 doesn't contain ã.

You can use code page 1252 from the console if you:

• Issue chcp 1252.
• Use a TrueType console font like Consolas or Lucida Console.
• Imbue the streams with English_United States.1252 instead.

Writing to a file has similar issues. If you view the file in Notepad, it uses the ANSI code page to interpret the bytes in the file. So even if a console app is using code page 437, Notepad will display the file incorrectly if written using the 437 code page. Writing the file in code page 1252 doesn't help either, because the two code pages don't interpret the same set of Unicode code points. Some answers to this problem are to get a different file viewer such as Notepad++ or write the file in UTF-8 which supports all Unicode characters.

Answer 3

You are having the opposite to the problem described here.

The core reason is the same: characters in the "ASCII"¹ range 128-256 are less standardized than the characters in the range 32-127. Most Windows applications, whether they use "Unicode" or "ANSI" strings, use the same mapping between codes and characters as specified by Unicode. however, for mostly historical reasons, the console uses a separate map of codes-to-characters usually called the "codepage". The exact table used depends of the language and configuration of Windows. For US English computers, that's the OEM 437 Code Page.

When you type ç in the console, you are really entering character code 135, because that's the code assigned to that character in the 437 code page used by the console. The rest of Windows interprets that character code as described in the Unicode tables as character ‡.

You can use OemToChar (documentation here) to convert text entered via the console to the corresponding string in Unicode encoding.

See my answer here for other background information.

---

¹ yes, this range is technical not ASCII, but close enough. I'm also using the usual informal (and technically wrong) definition of Unicode throughout.