Retrieve the content of a web page

Question

I'd like to fetch a webpage and save the content as a string? Is there a library to do that? I want to use the string for some a program I am building. It's for websites, that don't necessarily provide rss feed.

I can't possibly you believe you didn't find one. I don't believed you even began to search. [First Google result of 'java fetch webpage'](http://stackoverflow.com/questions/238547/how-do-you-programmatically-download-a-webpage-in-java) — Anti Earth, Jul 03 '13 at 16:00
@antiearth thanks, maybe the keyword i used to search was not accurate to the problem I was having. — , Jul 03 '13 at 16:05
@LuiggiMendoza It would be nice if you provide an example. Really. — giannis christofakis, Jul 03 '13 at 16:17
@yannishristofakis if you access to the links I've provided, there are lot of examples to accomplish the task asked by OP. Really. — Luiggi Mendoza, Jul 03 '13 at 16:37

score 3 · Accepted Answer · edited Dec 21 '16 at 16:13

3

i think you need this

URL url = new URL("http://www.google.com/");
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = null; // con.getContentEncoding(); *** WRONG: should use "con.getContentType()" instead but it returns something like "text/html; charset=UTF-8" so this value must be parsed to extract the actual encoding
encoding = encoding == null ? "UTF-8" : encoding;
String body = IOUtils.toString(in, encoding);
System.out.println(body);

edited Dec 21 '16 at 16:13

xav

5,452
7
48
57

answered Jul 03 '13 at 16:00

M Sach

33,416
76
221
314

Be careful, `con.getContentType()` should be used instead of `con.getContentEncoding()`, but it returns something like `"text/html; charset=UTF-8"` so this value must be parsed in order to extract the actual encoding (I've added a comment on the code above to reflect this) – xav Aug 21 '16 at 05:25
See http://stackoverflow.com/questions/5938007/what-is-the-difference-between-content-type-charset-x-and-content-encoding-x concerning my previous comment (`con.getContentEncoding()` is used for things like "gzip", "compress", ... not encoding) – xav Dec 21 '16 at 16:12

score 1 · Answer 2 · answered Jul 03 '13 at 16:03

1

May I suggest JSoup ?

Document doc = Jsoup.connect("www.google.com").get();

answered Jul 03 '13 at 16:03

merours

4,076
7
37
69

giannis christofakis · Answer 3 · 2014-05-29T15:05:17.390

0

You can use Apache HttpComponents

    CloseableHttpClient httpclient = HttpClients.createDefault();
    HttpGet httpget = new HttpGet("http://www.google.gr");
    try (CloseableHttpResponse response = httpclient.execute(httpget)) { 
        HttpEntity entity = response.getEntity();
        if (entity != null) {
           System.out.println(EntityUtils.toString(entity));
        }
        response.close();
    } catch (IOException ex) {
        Logger.getLogger(HttpClient.class.getName()).log(Level.SEVERE, null, ex);
    }

edited May 29 '14 at 15:05

answered Jul 03 '13 at 16:12

giannis christofakis

8,201
4
54
65

Hello. Do you know if this is slower or faster than the accepted answer? – dentex May 28 '14 at 18:47
@dentex I don't know. – giannis christofakis May 29 '14 at 15:04
@dentex I don't think you gain to much in performance. Don't forget you want to access a remote resource,so it's not up to you how fast your result will come,if speed is what concerns you. `Apache HttpComponents` gives you much more functionality like asynchronous calls. It's up to you. – giannis christofakis May 30 '14 at 13:46

Retrieve the content of a web page

3 Answers3