How to scrape .aspx site

Question

I'm trying to scrape: http://search.bgc-group.com/Default.aspx

When I'm running my code, I'm always getting the first page.

My code:

var casper = require('casper').create({
        verbose: true,
        logLevel: "debug",
        type: 'xpath',
    });
casper.start('http://search.bgc-group.com/Default.aspx');

casper.thenOpen('http://search.bgc-group.com/Default.aspx', {
    method: 'post',
    data:   {
        '__EVENTARGUMENT': 'Page$4',
        '__EVENTTARGET':  'ctl00$body$ctlJobListing1$gvJobListing',
        '__EVENTVALIDATION':  '',
        '__PREVIOUSPAGE':  '',
        '__VIEWSTATE':  ''
    }
});

casper.then(function() {
    this.echo('POSTED it.');
    this.capture("test.png")
});

casper.run();

I've not left the __EVENTVALIDATION, __PREVIOUSPAGE and __VIEWSTATE fields blank in the actual code. Just skipped here as they were very long string.

Where is the post data coming from? Why not use CasperJS for what it is intended for by filling that form and submitting it? — Artjom B., May 05 '14 at 08:41

score 1 · Accepted Answer · answered May 05 '14 at 08:57

1

You can do the following instead of the casper.thenOpen block:

var x = require('casper').selectXPath;
casper.thenClick("a[href*='Page$4']");
casper.waitForSelector(x("//td/span[text()='4']")); // might be unnecessary

answered May 05 '14 at 08:57

Artjom B.

61,146
24
125
222

1

Then you need specify it in your question, but it was already discussed at length see [here](http://stackoverflow.com/questions/23384963/casperjs-loop-or-iterate-through-multiple-web-pages) or [here](http://stackoverflow.com/questions/23291977/casperjs-parse-next-page-after-button-click) – Artjom B. May 05 '14 at 10:32

How to scrape .aspx site

1 Answers1