C# httpwebrequest and javascript

Question

I am using C# HttpWebRequest to get some data of a webpage. The problem is that some of the data is updated using javascript/ajax after the page is loaded and I am not getting it in the response string. Is there a way to have the webrequest wait untill all the scripts in the page have finished executing?

Thanks

Amit

score 17 · Answer 1 · answered Feb 05 '09 at 16:16

17

Just an idea but there is a way to have .net load a webpage as if it were in a browser: using System.Windows.Forms

you could Load the webpage into a WebBrowser control

WebBrowser wb = new WebBrowser();
wb.ScrollBarsEnabled = false;
wb.ScriptErrorsSuppressed = true;
wb.Navigate(url);
while (wb.ReadyState != WebBrowserReadyState.Complete) { Application.DoEvents(); }
wb.Document.DomDocument.ToString()

This will probably give you the pre ajax DOM but maybe there is a way to let it run the ajax first.

answered Feb 05 '09 at 16:16

missaghi

5,044
2
33
43

1

You will need to add a reference to System.Windows.Forms to access the WebBrowser class (if it's not already referenced in your project). – grasmi Mar 20 '13 at 11:48
1

Additionally, the following helped get a threaded version of this solution working. https://stackoverflow.com/questions/4269800/webbrowser-control-in-a-new-thread/4271581#4271581 – Jerrill May 26 '17 at 20:42

splattne · Accepted Answer · 2009-02-05T16:01:34.613

10

If I correctly interpret your question, there is no simple solution for your problem.

You are scraping the HTML from a server and since your C# code is not a real web browser, it doesn't execute client scripts.

This way you can't access information which the HTML you fetch doesn't contain.

Edit: I don't know how complex these AJAX calls from the original web site are, but you could use Firebug or Fiddler for IE to see how the requests are made in order to call these AJAX calls in your C# application too. So you could add the pieces of information you'll need. But it's only a theoretical solution.

edited Feb 05 '09 at 16:01

answered Feb 05 '09 at 14:16

splattne

102,760
52
202
249

I edited my question with a THEORETICAL solution... it depends on the circumstance, how often do the pages change... – splattne Feb 05 '09 at 14:23
I am checking the data every 30 minutes. I think thats what i will have to do, bummer! – Amit Raz Feb 05 '09 at 14:25
Generally you would run a real browser and automate it from C#. https://www.seleniumhq.org/ – ed22 Jul 30 '19 at 07:16

score 4 · Answer 3 · answered Jul 22 '10 at 12:13

4

Use HttpWebRequest to download the page, programatically search the source code for the relevant ajax information and then use a new HttpWebRequest to pull that data down.

answered Jul 22 '10 at 12:13

Chris Almond

577
3
6

score 4 · Answer 4 · answered Feb 05 '09 at 14:16

When you open a web page in a web browser, it is the browser that executes the javascript and downloads additional resources used by the page (images, scripts, etc). HttpWebRequest by itself will not do any of this, it will only download the html for the page you requested. It will never execute any of the javascript/ajax code on it's own.

score 3 · Answer 5 · answered Feb 05 '09 at 14:18

3

HttpWebRequest does not emulate a web browser, it just downloads the resource you point it at. This means it will not execute or even download JavaScript files.

You would have to use something like FireBug to get the URL for the data being pulled in via JavaScript, and point your HttpWebRequest at that.

answered Feb 05 '09 at 14:18

roryf

29,592
16
81
103

I thought of that but it is not a URL its a number and when i use the http request the label which the number is on is empty – Amit Raz Feb 05 '09 at 14:22
I had the same idea. But I guess it will be a PITA to keep that code working over time... – splattne Feb 05 '09 at 14:24
from the 30 minutes I have tryed, doing it this way seems not simple at all. – Amit Raz Feb 05 '09 at 14:44

score 1 · Answer 6 · edited May 24 '12 at 20:48

1

Use HttpWebRequest to download the page. Search the source code for the relevant AJAX information and then use a new HttpWebRequest to pull that data down.

edited May 24 '12 at 20:48

Bob Kaufman

12,864
16
78
107

answered Dec 24 '10 at 12:05

Nabin Aryal

31
2

Such data has to be executed, you can't solve problem by just getting it – Stefan Đorđević Aug 25 '19 at 02:10

score 0 · Answer 7 · edited May 23 '17 at 10:31

You could use of the PhantomJs. I had this Issue, but don't found solution for my problem. In my opinion, best solution is This.

My solution is look like this:

var page = require('webpage').create();

page.open("https://sample.com", function(){
    page.evaluate(function(){
        var i = 0,
        oJson = jsonData,
        sKey;
        localStorage.clear();

        for (; sKey = Object.keys(oJson)[i]; i++) {
            localStorage.setItem(sKey,oJson[sKey])
        }
    });

    page.open("https://sample.com", function(){
        setTimeout(function(){
         page.render("screenshoot.png") 
            // Where you want to save it    
           console.log(page.content); //page source
            // You can access its content using jQuery
            var fbcomments = page.evaluate(function(){
                return $("body").contents().find(".content") 
            }) 
            phantom.exit();
        },10000)
    });     
});

C# httpwebrequest and javascript

7 Answers7

Linked