0

I'm trying to download stock option data from Yahoo Finance (here's Google as an example) with requests.get, which doesn't seem to be downloading everything. I'm trying to get the dropdown of dates with an XPath but even //option doesn't return anything even though Chrome DevTools says there are 13 instances!

I expect this has something to do with the fact that the parts of the site that actually matter are being loaded after all the navigation bars and such, and I don't know how to get all of it. Could you please suggest a method for getting the text of each item in the date dropdown menu?

spelchekr
  • 933
  • 3
  • 11
  • 19
  • http://stackoverflow.com/questions/39218742/using-beautifulsoup-to-search-through-yahoo-finance – Rafael Oct 13 '16 at 11:53

1 Answers1

0

If you open the dev console and refresh the page again (caches might need to be purged), you can see some requests with type xhr.

They are usually initiated by JavaScript programs and will load some data besides those provided by HTML body.

That's what you can look into.

Zixian Cai
  • 945
  • 1
  • 10
  • 17
  • Just to tag onto this answer: https://query1.finance.yahoo.com/v7/finance/options/GOOG?formatted=true&crumb=B2JsfXH.lpf&lang=en-US&region=US&corsDomain=finance.yahoo.com see if that link has the information you want. – Rafael Oct 07 '16 at 04:10
  • In Incognito Mode (which I understand to use a new cache and everything), the console still doesn't give me any `xhr` requests... This is the console that's the second tab in the Chrome devtools, right? – spelchekr Oct 07 '16 at 17:03
  • @RafaelMartins Could you elaborate on how you got that link please? – spelchekr Oct 13 '16 at 03:09