7

I just want the text content of page and I want the fetching to be as lightweight as possible. Can I turn off all the parsing and additional loading of JavaScript, CSS and other external content that HTMLUnit does out of the box?

Mosty Mostacho
  • 42,742
  • 16
  • 96
  • 123
Thomas
  • 10,289
  • 13
  • 39
  • 55

1 Answers1

12

I think the closest thing to what you're looking for is:

WebClient webClient = new WebClient();
webClient.setCssEnabled(false);
webClient.setAppletEnabled(false);
webClient.setJavaScriptEnabled(false);

For HtmlUnit 2.13 and above, use webclient.getOptions().

Also this question and answer might be useful too. It really made things faster for me, but I had to recompile HtmlUnit...

Finally, in order to get the original content of the page (instead of the output of asXml()) try the following:

WebClient webClient = new WebClient();
HtmlPage page = webClient.getPage("http://www.yourpage.com");
String originalHtml = page.getWebResponse().getContentAsString();
Community
  • 1
  • 1
Mosty Mostacho
  • 42,742
  • 16
  • 96
  • 123
  • 1
    Thank you for your answer. What would be the difference between `asXML()` and `page.getWebResponse().getContentAsString()` ? – Thomas Apr 11 '12 at 09:22
  • 2
    `asXML()` will format the code (for instance, add spaces whenever an html tag is opened) while `getContentAsString()` will show you the html code exactly as it is returned by the web server – Mosty Mostacho Apr 11 '12 at 15:55
  • 3
    Starting HTMLUnit 2.13, use webClient.getOptions() to invoke these "enable" methods. – Paddy Mar 12 '14 at 15:47
  • @Mosty Is there a specific reason you are disabling applets too. – Chiseled Jun 26 '14 at 16:46