How to send Post Data to https server via HttpsURLConnection

Question

I saw a form (https://aptransport.in/CFSTONLINE/Reports/VehicleRegistrationSearch.aspx) in web, if i give Select Search Element: as Registration No and Enter Search Element: as AP31BF2942 ,and if i click on get data button, then i am getting my vehicle Details.
I want to do this in HttpsURLConnection.
I have seen parameter names as ctl00$OnlineContent$ddlInput and ctl00$OnlineContent$txtInput in the source of url.

But i am unable to get required data.Please check the url

import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

import javax.net.ssl.HttpsURLConnection;

public class TestAPHttp {

private final String USER_AGENT = "Mozilla/5.0";

public static void main(String[] args) throws Exception {

    TestAPHttp http = new TestAPHttp();


    System.out.println("\nTesting  - Send Http POST request");
    http.sendPost();

}



// HTTP POST request
private void sendPost() throws Exception {

    String url = "https://aptransport.in/CFSTONLINE/Reports/VehicleRegistrationSearch.aspx";
    URL obj = new URL(url);
    HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();

    //add reuqest header
    con.setRequestMethod("POST");
    con.setRequestProperty("User-Agent", USER_AGENT);
    con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");

    String urlParameters = "ctl00$OnlineContent$ddlInput=R&ctl00$OnlineContent$txtInput=AP31BF2942";

    // Send post request
    con.setDoOutput(true);
    DataOutputStream wr = new DataOutputStream(con.getOutputStream());
    wr.writeBytes(urlParameters);
    wr.flush();
    wr.close();

    int responseCode = con.getResponseCode();
    System.out.println("\nSending 'POST' request to URL : " + url);
    System.out.println("Post parameters : " + urlParameters);
    System.out.println("Response Code : " + responseCode);

    BufferedReader in = new BufferedReader(
            new InputStreamReader(con.getInputStream()));
    String inputLine;
    StringBuffer response = new StringBuffer();

    while ((inputLine = in.readLine()) != null) {
        response.append(inputLine);
    }
    in.close();

    //print result
    System.out.println(response.toString());

}

}

Answer 1

Instead of

DataOutputStream wr = new DataOutputStream(con.getOutputStream());

it should read

OutputStream wr = con.getOutputStream();

When converting a String to byte[] you should set the encoding:

wr.write(urlParameters.getBytes("UTF-8"));

And you should set the Content-Type header:

con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");

BTW you're "eating" the line endings of the received document. readLine() reads the next characters until line ending, but doesn't return \n to you. You should append them to your response. And in almost any case you shouldn't use StringBuffer, but StringBuilder, but this has absolutely nothing to do with this post and is only for your information.

Edit

I tried it in the meanwhile with my HTTP client library DavidWebb and I set some additional headers like User-Agent and all missing form params and now it works:

Webb webb = Webb.create();
Response<String> response = webb
        .post("https://aptransport.in/CFSTONLINE/Reports/VehicleRegistrationSearch.aspx")
        .header("User-Agent",
                "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/34.0.1847.116 Chrome/34.0.1847.116 Safari/537.36")
        .header("Accept", "*/*")
        .header("Origin", "https://aptransport.in")
        .header("Referer",
                "https://aptransport.in/CFSTONLINE/Reports/VehicleRegistrationSearch.aspx")
        .header("X-MicrosoftAjax", "Delta=true")
        .param("ctl00$OnlineContent$ddlInput", "R")
        .param("ctl00$OnlineContent$txtInput", "AP31BF2942")
        .param("ctl00$OnlineContent$ScriptManager1",
                "ctl00$OnlineContent$Updatepanel1|ctl00$OnlineContent$btnGetData")
        .param("__EVENTTARGET", "")
        .param("__EVENTARGUMENT", "")
        .param("__VIEWSTATE",
                "/wEPDwUKMTE0MzI5ODM0MGRkdB0g7u2Z+Eg+a4Wr8QLkE1lzgSU=")
        .param("ctl00$OnlineContent$btnGetData", "Get Data")
        .connectTimeout(3000)
        .asString();

if (response.isSuccess()) {
    String result = response.getBody();
    System.out.println(result);
} else {
    System.out.println(response.getStatusCode());
    System.out.println(response.getResponseMessage());
    System.out.println(response.getErrorBody());
}

There are many cases, where simulating a browser and using HttpURLConnection doesn't work or is very complicated (Cookies, client-side JavaScript, Timing, special headers, Cross-Site-Request-Forgery protection, ...). In your case, you have luck, but small changes in the application might break your working code while you're sleeping.

So how did I solve the problem? I analyzed the network traffic (sent headers and form data) with Chrome's developer tools (you can as well use FireFox, Firebug, ...) but in case of HTTPS it should be integrated in the browser - you can't sniff the SSL encrypted traffic with wireshark e.g.

You can see in the request, that many more headers and form parameters are sent across the wire: