Submit a query automatically with cheerio (like JQuery)

Question

I have to do the following: given a page loaded from a server, I have to submit a form in that page automatically after loading it, filling up all of its input fields with the same string (test worked). This is obviously a simplification of my problem, but it's close enough!

At the moment I am getting the form, and using https://github.com/cheeriojs/cheerio to explore it:

var URL = http://www.example.com/form/index.html"
// ...load the page ...
$ = cheerio.load( fetchedPageText );

At the moment, I have the following code:

var $ = cheerio.load( fetchedPageText );

var forms = $('form');
for( var i1 = 0, l1 = forms.length; i1 < l1; i1 ++ ){
  var form = forms[ i1 ];

  inputFields = $( 'input', form );


  console.log("******FORM ACTION: ", form.attribs.action );
  console.log("******FORM: ", form );

  for( var i2 = 0, l2 = inputFields.length; i2 < l2; i2 ++ ){
    var inputField = inputFields[ i2 ];

    console.log( inputField );
    console.log("**************INPUT FIELD ", inputField );


    /* At this point, I have `action` and every input field */


  }
};

Questions:

• At the moment, submit is relative to the page I have downloaded. How do I make sure that I submit things in the right spot? Should I do url.parse and work out the gull path for the action from the URL?

• How do you actually create a "post" string? Or, even better, how would you post this form?

• I realise that this might not work (the form might have Javascript, etc.). However, is there anything else I need to be careful about, when submitting this form?

Answer 1

To post the form to the correct address you must combine page's base url and the one in the post attribute. url.resolve(from, to) can help

To post the form you can use e.g. http://visionmedia.github.io/superagent/ or some other ajax library with support for application/x-www-form-urlencoded

You should be aware that many sites apply various anti-spam measures and they will reject your requests if not properly formed. It must be checked on per-site basis.

You should also be aware that mass mailing or mass form submitting is in some countries prosecuted by law.

You should be aware that there are usually much easier APIs to be used by 3rd parties (other then parsing html forms) based on JSON and REST or SOAP.

If you need to submit pages to a web server legally better option would be to negotiate another API with the server owner. It should not be a problem even with government servers as many of them are opening due to the Open Data initiative (America, Europe, India, ..)