Get Generated HTML On Android

Question

I'm making an app that take some data from a website using regex. But those data are loaded only after the complete page is loaded. So the source code I get only contain the script that call that code. Is there any way I can get that generated html source code.

The data that I mentioned is actually google search result inside a website. When I give search query through the url like

website.com/search?q=car

The result gives all links inside that site related to the query and I'm trying to extract link urls from the result.

I believe this code in that page generate the html script.

<script>
    (function() {
        var cx = '013305635491195529773:0ufpuq-fpt0';
        var gcse = document.createElement('script');
        gcse.type = 'text/javascript';
        gcse.async = true;
        gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') +
        '//www.google.com/cse/cse.js?cx=' + cx;
        var s = document.getElementsByTagName('script')[0];
        s.parentNode.insertBefore(gcse, s);
    })();
    </script>

It's easier to help you if we can see what code you currently have. My current recommendation without looking at your code is to try to call your script after the page has loaded, by using `window.onload` or if you use JQuery by using `$(document).ready(...);` I'm assuming you are getting the result using javascript. — alfredo, Apr 28 '17 at 16:32
The problem is that , the site is not in my control. I can do changes from android side. Is there any way to load page only after complete script is executed.. — pz64_, Apr 28 '17 at 16:49
Oh, I understand now. Are you creating a native app using Java and Android Studio, or are you using something like Cordova/Phonegap or cocos2d-x or Titanium? — alfredo, Apr 28 '17 at 18:23
If you are using Android Studio look at this solution (http://stackoverflow.com/questions/2376471/how-do-i-get-the-web-page-contents-from-a-webview/4892013#4892013). — alfredo, Apr 28 '17 at 19:08
I believe you took me to right place..I'll notify if its working..Let me check... — pz64_, Apr 28 '17 at 19:32
Wooha..its working...Loading url into webview (with javascript turned on) and creating a JavaScriptInterface to read the code.. — pz64_, Apr 29 '17 at 05:14
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/142967/discussion-between-alfredo-and-pzy64). — alfredo, Apr 29 '17 at 14:35

score 3 · Accepted Answer · edited May 23 '17 at 11:46

The solution that worked for the OP in Android Studio to get the source code of an external website after Javascript had run is the code from this other Stackoverflow question How do I get the web page contents from a WebView?

final Context myApp = this;

/* An instance of this class will be registered as a JavaScript interface */
class MyJavaScriptInterface
{
    @JavascriptInterface
    @SuppressWarnings("unused")
    public void processHTML(String html)
    {
        // process the html as needed by the app
    }
}

final WebView browser = (WebView)findViewById(R.id.browser);
/* JavaScript must be enabled if you want it to work, obviously */
browser.getSettings().setJavaScriptEnabled(true);

/* Register a new JavaScript interface called HTMLOUT */
browser.addJavascriptInterface(new MyJavaScriptInterface(), "HTMLOUT");

/* WebViewClient must be set BEFORE calling loadUrl! */
browser.setWebViewClient(new WebViewClient() {
    @Override
    public void onPageFinished(WebView view, String url)
    {
        /* This call inject JavaScript into the page which just finished loading. */
        browser.loadUrl("javascript:window.HTMLOUT.processHTML('<head>'+document.getElementsByTagName('html')[0].innerHTML+'</head>');");
    }
});

/* load a web page */
browser.loadUrl("http://lexandera.com/files/jsexamples/gethtml.html");

score -1 · Answer 2 · answered Apr 28 '17 at 14:34

-1

You can use WebView. Get the data from url, then save it in some variable. https://developer.android.com/reference/android/webkit/WebView.html

answered Apr 28 '17 at 14:34

rew1nd

33
7

Get Generated HTML On Android

2 Answers2