6

I have a web site with forms that I need to scrape. Instead of filling the flash forms, I would like to POST some keys/values to the URL that doesn't support GET requests.

I use spynner to interact with the site, and spynner can have a GUI, but my search on google, stackoverflow, spynner github and in the spynner module are unsuccessful.

If spynner can't do a POST request, maybe gtk or qt + webkit can do that ? Any real life code sample will be really appreciated.

Gilles Quénot
  • 173,512
  • 41
  • 224
  • 223
  • 1
    Qt definitely has that feature. Have a look at [QWebFrame#load](http://doc.qt.nokia.com/4.7-snapshot/qwebframe.html#load-2) and [QNetworkRequest](http://doc.qt.nokia.com/4.7-snapshot/qnetworkrequest.html) – Niklas B. May 12 '12 at 13:41

1 Answers1

5

You can do it like this with Spynner:

import spynner
from PyQt4.QtCore import QUrl
from PyQt4.QtNetwork import QNetworkRequest, QNetworkAccessManager

url = "http://localhost:8080/niklas/test.php"
data = "foo=bar"
headers = { "Content-Type": "application/x-www-form-urlencoded" }

req = QNetworkRequest(QUrl(url))
for k, v in headers.items():
    req.setRawHeader(k, v)

browser = spynner.Browser()
browser.webframe.load(req, QNetworkAccessManager.PostOperation, data)
browser._wait_load()

print browser.html
Niklas B.
  • 92,950
  • 18
  • 194
  • 224
  • 2
    And if we would like to `POST` one more time in the same script, what is the syntax ? If I try to run `browser.webframe.load()` one more time, I get the following error : "QNetworkReplyImplPrivate::error: Internal problem, this method must only be called once." – Gilles Quénot May 13 '12 at 12:46
  • 1
    @sputnick: I get this error only if I run `load` twice in a row without waiting. Did you call `_wait_load()` in between? – Niklas B. May 13 '12 at 12:49
  • @GillesQuenot Spynner is based partly on Webkit. [Webkit bug 82506](https://bugs.webkit.org/show_bug.cgi?format=multiple&id=82506) discusses this behavior. – MarkDBlackwell Oct 20 '15 at 11:52