C# To transform Facebook Response to proper encoded string

Question

I am using regular Stream Reader to get response from Facebook graph API response https://graph.facebook.com/XXXX?access_token=&fields=id,name,about,address,last_name

I am reading the response stream yet it returns me {"id":"XXXXX","name":"K\u0131r\u0131nt\u0131 Reklam"...}

My code is below - I unsuccessfully tried using explicitly UTF-8 and "iso-8859-9" (Turkish) encodings and setting accept-charset headers. I read Joel's famous article about encodings. It looks like each of the chars '\' 'u' '1' '3' '1' are coming as characters from facebook - I thought this would have been 2 bytes for value 131 in UTF-8. I am confused. I expect this string to be "Kırıntı Reklam".

I could simply find/replace those strings - yet it would be far from elegant and maintainable. How should I properly process or convert the facebook graph api response for strings with accents?

using (WebResponse response = request.GetResponse())
{
using (Stream dataStream = response.GetResponseStream())
{
    if (dataStream != null)
    {
        using (StreamReader reader = new StreamReader(dataStream))
        {
            responseFromServer = reader.ReadToEnd();
        }

    }
}
}

Thank you in advance

Answer 1

tldr; use a JSON library - I like Json.NET - and don't worry about it.

The JSON shown is valid JSON where \uABCD in a JSON string represents a UTF-16 encoded character¹. The internal JSON character escaping format is useful to avoid having to deal with Unicode stream encoding issues - it allows JSON to be represented entirely in ASCII/7-bit-clean characters (which is a subset of UTF-8).

Using a conforming JSON library to parse the JSON with such escapes would restore the JSON into an appropriate object-graph, of which some values will be properly-decoded String values. The library is responsible for understanding JSON and converting/reading it as appropriate - this includes correctly handling any such \u escape sequences.

The stream itself (that of the JSON text) should use the encoding that the server says, is indicated by a BOM, or has been pre-negotiated: but really, just UTF-8 here. This is how the JSON text is encoded, but has no bearing on the escape sequences found in JSON strings.

---

¹ Per RFC 4627, The application/json Media Type for JavaScript Object Notation (JSON):

Any character may be escaped. If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the character's code point. The hexadecimal letters A though F can be upper or lowercase. So, for example, a string containing only a single reverse solidus character may be represented as "\u005C".

Alternatively, there are two-character sequence escape representations of some popular characters. So, for example, a string containing only a single reverse solidus character may be represented more compactly as "\\".

To escape an extended character that is not in the Basic Multilingual Plane, the character is represented as a twelve-character sequence, encoding the UTF-16 surrogate pair. So, for example, a string containing only the G clef character (U+1D11E) may be represented as "\uD834\uDD1E"

---

For the doubters, here is a LINQPad example. This uses JSON.Net and imports the Newtonsoft.Json.Linq namespace.

var json = @"{""name"":""K\u0131r\u0131nt\u0131 Reklam""}";
json.Dump();                        // -> {"name":"K\u0131r\u0131nt\u0131 Reklam"}
var name = JObject.Parse(json)["name"].ToString();
(name == "Kırıntı Reklam").Dump();  // -> true