I'm investigating subj, as there's one site where a part of data is accessible after clicking on a said link.
I've a script like this:
#!/usr/bin/env python
script="""
splash:go(splash.args.url)
splash:wait(10)
splash:runjs("$('a[data-bet-type=FixedPlace]')[0].click()")
splash:wait(10)
return {
-- html = splash:html(),
png = splash:png(),
-- har = splash:har(),
}
"""
from requests import post
from json import dumps, dump
from base64 import b64decode
from contextlib import suppress
if 1:
endpoint='http://localhost:8050/run'
j={
'lua_source': script,
'url': 'https://www.odds.com.au/horse-racing/bunbury/race-1',
'timeout': 90,
'headers': {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0'
}
}
r=post(endpoint, headers={'Content-Type': 'application/json'}, data=dumps(j))
print(r.status_code)
if r.status_code != 200:
print(r.text)
exit()
j=r.json()
for _ in ('html', 'har'):
with suppress(KeyError), open('1.'+_, 'w') as f: f.write(j[_])
with suppress(KeyError), open('1.png', 'wb') as f: f.write(b64decode(j['png']))
if 'har' in j:
for entry in j['har']['log']['entries']:
url=entry['request']['url']
if url.startswith('https://www.odds.com.au'): print(url)
But although it's perfectly rendering the page, the click doesn't happen. I've tried with ya.ru and the same approach worked (but it was a button in that case). I'm out of ideas. Tried setting UA, playing with waits (it's not fast, that page), using js_source, but cannot click a thing, although I see the element is found.
