JSON URL from StackExchange API returning jibberish?

Question

I have a feeling I'm doing something wrong here, but I'm not quite sure if I'm missing a step, or am just having an encoding problem or something. Here's my code:

URL url = new URL("http://api.stackoverflow.com/0.8/questions/2886661");

   BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
   // Question q = new Gson().fromJson(in, Question.class);
   String line;
   StringBuffer content = new StringBuffer();
   while ((line = in.readLine()) != null)
   {
    content.append(line);
   }

When I print content, I get a whole bunch of wingdings and special characters, basically jibberish. I would copy and past it here, but that isn't working. What am I doing wrong?

Answer 1

In this case it's not a character encoding problem, it's a content encoding problem; you're expecting text, but the server is using compression to save bandwidth. If you look at the headers when you grab that url, you can see the server you are connecting to is returning gzipped content:

GET /0.8/questions/2886661 HTTP/1.1
Host: api.stackoverflow.com

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 22 May 2010 15:51:34 GMT
Content-Type: application/json; charset=utf-8
<more headers>
Content-Encoding: gzip
<more headers>

So you either need to use a smarter client like Apache's HttpClient as stevedbrown suggests (although you need a tweak to get it to speak Gzip automatically), or explicitly decompress the stream you got in your example code. Try this instead for the line where you declare your input:

 BufferedReader in = new BufferedReader(new InputStreamReader(new GZIPInputStream(url.openStream())));

I've verified that this works for the url you are trying to grab.

Answer 2

Sometimes the API call response are compressed eg. StackExchange API. Please go through their documentation and check for the compression they are using. Some use either GZIP or DEFLATE compression.In case of GZIP compression use the following.

InputStream is = new URL(url).openStream();
BufferedReader in = new BufferedReader(new InputStreamReader(new GZIPInputStream(is)));

Answer 3

Use the Apache Http Client instead, it's going to take care of character conversions properly. From that site's examples:

public final static void main(String[] args) throws Exception {

    HttpClient httpclient = new DefaultHttpClient();

    HttpGet httpget = 
        new HttpGet("http://api.stackoverflow.com/0.8/questions/2886661"); 

    System.out.println("executing request " + httpget.getURI());

    // Create a response handler
    ResponseHandler<String> responseHandler = new BasicResponseHandler();
    String responseBody = httpclient.execute(httpget, responseHandler);
    System.out.println(responseBody);

    System.out.println("----------------------------------------");

    // When HttpClient instance is no longer needed, 
    // shut down the connection manager to ensure
    // immediate deallocation of all system resources
    httpclient.getConnectionManager().shutdown();        
}

In this case, see http://svn.apache.org/repos/asf/httpcomponents/httpclient/branches/4.0.x/httpclient/src/examples/org/apache/http/examples/client/ClientGZipContentCompression.java, which shows how to deal with Gzip content.