0

I have a premium account with morningstar and I tried to download a few csv files from the premium content area. For some reason I cannot get those premium content. For example, with premium account I can get 10 year financial statement data, but I've tried all the sample authentication java code from apache httpcomponents-client. All of them can only get me content that does not need authentication. How can I tell what authentication protocol morningtar is using and authenticate successfully? I tried the example code from org.apache.http.examples.client, including clientAuthentication.java, clientKerberosAuthentication.java, clientInteractiveAuthentication.java . If I log in in morningstar account in Chrome and paste this URL, I can get 10 years data csv, but if I access through java I only get 5 years data. Below are one of sample codes I tried. I didn't get exceptions or errors, but I only got 5 years data instead of 10.

import java.io.BufferedReader;
import java.io.InputStreamReader;

import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;


public class ClientAuthentication {

    public static void main(String[] args) throws Exception {
        DefaultHttpClient httpclient = new DefaultHttpClient();
        try {
            httpclient.getCredentialsProvider().setCredentials(
                    new AuthScope("morningstar.com", 443),
                    new UsernamePasswordCredentials("xxx@gmail.com", "xxxx")); //anonymized this before posting to stackoverflow

            HttpGet httpget = new HttpGet("http://financials.morningstar.com/ajax/ReportProcess4CSV.html?t=aapl&region=usa&culture=en_US&reportType=is&period=12&dataType=A&order=asc&columnYear=10&rounding=3&view=raw&productCode=USA&r=199209&denominatorView=raw&number=3");

            System.out.println("executing request" + httpget.getRequestLine());
            HttpResponse response = httpclient.execute(httpget);
            HttpEntity entity = response.getEntity();
            BufferedReader in;
            in = new BufferedReader(new InputStreamReader(entity.getContent()));

            System.out.println("----------------------------------------");
            System.out.println(response.getStatusLine());
            if (entity != null) {
                System.out.println("Response content length: " + entity.getContentLength());
                int linenum = 0;
                while (true){
                    String line = in.readLine();
                    if (line == null) break;
                    linenum++;  
                    if (linenum>1)
                    System.out.println(line);

                }
            }
            EntityUtils.consume(entity);
        } finally {
            // When HttpClient instance is no longer needed,
            // shut down the connection manager to ensure
            // immediate deallocation of all system resources
            httpclient.getConnectionManager().shutdown();
        }
    }
}
user1030532
  • 573
  • 1
  • 5
  • 12
  • What have you tried? What errors are you getting? We need to see code and such. – CodeChimp Mar 29 '13 at 14:10
  • I tried the example code from org.apache.http.examples.client, including clientAuthentication.java, clientKerberosAuthentication.java, clientInteractiveAuthentication.java . The url to csv file is http://financials.morningstar.com/ajax/ReportProcess4CSV.html?t=aapl&region=usa&culture=en_US&reportType=is&period=12&dataType=A&order=asc&columnYear=10&rounding=3&view=raw&productCode=USA&r=199209&denominatorView=raw&number=3. If I log in in Chrome and paste this URL, I can get 10 years data csv, but if I access through java I only get 5 years data. thanks – user1030532 Mar 29 '13 at 14:20
  • Can you update your question with any relavent code, exceptions, configs, logs available (please wrap them as Code using the {} button). It would greatly improve your chances of getting a solid answer. – CodeChimp Mar 29 '13 at 14:22
  • just updated my question with one sample code i tried as CodeChimp suggested – user1030532 Mar 29 '13 at 15:27
  • What error are you getting in the above code? You may need to to somehow either simulate a browser logging in, or using something like Webkit (or the Java equivalent) to interface with the site. I would also recommend checking out this: http://stackoverflow.com/questions/3283234/http-basic-authentication-in-java-using-httpclient – CodeChimp Mar 29 '13 at 16:20
  • I didn't get any error. – user1030532 Mar 29 '13 at 16:39
  • You may not have gotten an exception, but if you didn't get the file you were after, you got an error. You probably got an HTTP error, and will have to print out the result to see it. – CodeChimp Mar 29 '13 at 17:11
  • I tried basic authentication it didn't work, still got me the result same as without authentication. – user1030532 Mar 29 '13 at 17:12
  • what HTTP properties should I print out to see possible errors? – user1030532 Mar 29 '13 at 17:18
  • Basic troubleshooting: 1) Does my input look good? 2) Does the output look good? You are printing the URL prior to request, but I don't see where you are printing your output (http headers, response body, etc). I am not familar with Apache HttpClient, and would have to dig through the APIs to determin how exactly to output all of that, but basically you want to see what the web server you are requesting too replied with. It's possible the Authentication worked, and the server's response was a 302 redirect, meaning you need to make a subsequent request to another URL, or some other code. – CodeChimp Mar 29 '13 at 17:25
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/27196/discussion-between-user1030532-and-codechimp) – user1030532 Mar 29 '13 at 17:55

0 Answers0