Mechanize and invisible search form

Question

I'm trying to perform search on some website using Mechanize but I can't submit a search form because mechanize does not see any forms. page.form returns nil and page = agent.get returns just {forms}> while I expect something like

<Mechanize::Form
 {name "somename"}
 {method "GET"}
 {action "/search"}

Is it because the search form uses javascript? Is there any way to solve this? Or the only way is to give up on mechanize and use something else?

Mechanize cannot do javascript. http://stackoverflow.com/questions/802225/how-do-i-use-mechanize-to-process-javascript — Unixmonkey, Dec 17 '12 at 22:04
Could you post the URL to the page you're trying to crawl? Unixmonkey is absolutely right. You can also use a webdriver (like waitir) to load the page first. — sunnyrjuneja, Dec 17 '12 at 22:26
This is the page http://s.weibo.com/ I knew that mechanize had problems with js but I was not sure if this specific problem was because of js. And I hoped that there's possible some workaround... For mechanize seems to be perfect for my needs and the idea of using something else makes me sad) — hiirulainen, Dec 17 '12 at 22:56

score 0 · Accepted Answer · answered Dec 18 '12 at 01:24

It means there's no form on that page. The workaround is to get the next page, the one that's pretending to be a form submit.

In other words when I type 'foo' into the search box and click the button I get redirected to: http://s.weibo.com/weibo/foo&Refer=index So just get that page and do something with it.

Mechanize and invisible search form

1 Answers1