Java Passing URLConnection Parameter when query webpage

Question

I would like to carry out searches on www.domain.com.au where a number of URL parameters need to be supplied in order to get a successful HTTP response. However, the

  1.  String url = "http://www.domain.com.au/"; // or www.realestate.com.au
  2.  String charset = "UTF-8";
  3.  String param1 = "Melbourne 3000 VIC"; // simple search on the web page
  4.
  5.  URLConnection connection = (URLConnection) new URL(url + "?" + param1).openConnection();
  6.  connection.setRequestProperty("Accept-Charset", charset);
  7.  connection.connect();
  8.  connection.setDoInput(true);
  9.  connection.setDoOutput(false);
 10.  connection.connect();
      .......

The response was a HTTP 400 error. As a result, I need your advice on the following questions to understand where the issue is coming from:

( i ) Should the parameter name be param1, or something else on the web page? If so, what is it?

( ii ) Is line 5 correct assuming that the name of the parameter is param1? Is it necessary to include URLEncoder.encode(param1, charset) even though line 6 has already set it explicitly.

The above code snippet works on other simple webpage where no parameters are required.

Your advice would be much appreciated.

Thanks,

Jack

score 0 · Answer 1 · edited May 23 '17 at 12:26

0

Using java.net.URLConnection is described very well in the following link by BalusC

Using java.net.URLConnection to fire and handle HTTP requests

edited May 23 '17 at 12:26

Community

1
1

answered Aug 31 '12 at 08:21

Mehdi

4,396
4
29
30

Thanks BalusC for offering your advice on this thread. I have read this article quite a number of times and use the example in it in this thread but it did not work for me, mostly likely due to my lack of knowledge & interpretation of what this article was referring to. It would be fantastic if you could lookup the same website to understand what the correct URL including parameter as well. Thanks, Jack – George Smith Sep 01 '12 at 08:23

score 0 · Answer 2 · edited May 23 '17 at 10:24

0

Couple of things:

1 . There you are not passing a name value pair. A typical GET URL request looks like

http://www.domain.com.au/?param1=value

you are just passing value and your URL looks like

http://www.domain.com.au/?Melbourne 3000 VIC

2 . Change line no. 5 to

URLConnection connection = (URLConnection) new URL(url + "?parame1=" + param1).openConnection();

Additionally encode the query string to substitute the spaces with html encode. Check here how to do that.

edited May 23 '17 at 10:24

Community

1
1

answered Aug 31 '12 at 08:21

Santosh

17,667
4
54
79

Thank you for the update. However, suggestion 1. failed to even connect to www.domain.com.au & ended up at Google search prompting for clarification. On the other hand, the correct URsL should be something like the following that returned valid results: – George Smith Aug 31 '12 at 14:32
Thank you for the update. However, suggestion 1. failed to even connect to www.domain.com.au & ended up at Google search prompting for clarification. On the other hand, the correct URsL should be something like the following that returned valid results: http://www.domain.com.au/Search/buy/State/VIC/Area/Inner-City/Region/Melbourne-Region/Suburb/Melbourne/?ssubs=1&searchterm=Melbourne Are you able to lookup this web page to clarify which valid URL including parameter should I use? Thanks again, Jack – George Smith Aug 31 '12 at 14:45

score 0 · Answer 3 · answered Jul 29 '14 at 00:03

I have checked the doamin.com.au website.

1) The name of the search parameter is "searchterm" and missed the key(key=value) in the URL.

HttpURLConnection connection = (HttpURLConnection) new URL(url + "?searchterm=" + param1).openConnection();

2) You should include encoding if there are special characters.

Here are the code example which will return the content from the website:

    public void searchDomain(String key) throws Throwable
{
    String url = "http://www.domain.com.au/"; // or www.realestate.com.au
    String charset = "UTF-8";
    String param1 = "Melbourne"; // simple search on the web page
    HttpURLConnection connection = (HttpURLConnection) new URL(url + "?searchterm=" + param1).openConnection();
    connection.setRequestProperty("Accept-Charset", charset);
    connection.setDoInput(true);
    connection.setDoOutput(true);
    connection.connect();

    java.io.BufferedReader br = new java.io.BufferedReader(new java.io.InputStreamReader(connection.getInputStream()));
    java.lang.StringBuffer sb = new java.lang.StringBuffer();
    java.lang.String str = br.readLine();
    while (str != null) 
    {
        sb.append(str);
        str = br.readLine();
    }
    br.close();
    connection.disconnect();

    System.out.println(sb.toString());
}

Java Passing URLConnection Parameter when query webpage

3 Answers3