How to read from the network tab in Python?

Question

Using Selenium, Requests, & Beautiful Soup, I'd like to be able to locate & print this .m3u8 link (or any of the links shown here) but I don't know to show requests in Python.

def locator(url):
    driver = sp.driver # just geckodriver with profile
    driver.get(url)
    sleep(4)
    # from here needs to somehow access the network tab & locate GET requests with Host == "cfvod.kaltura.com"

score 1 · Accepted Answer · answered Nov 04 '20 at 06:53

Had to use ChromeDriver, PyChrome, and The DevTools Protocol, but this worked:

def outputstart(**kwargs):
    print("START ", kwargs)

driver = sp.driver # my chromedriver profile with an argument added for port 8000

dev_tools = pychrome.Browser(url="http://localhost:8000")
tab = dev_tools.list_tab()[0]
tab.start()

url = 'https://google.com'

start = time.time()
driver.get(url)
tab.call_method("Network.emulateNetworkConditions",
            offline=False,
            latency=100,
            downloadThroughput=93750,
            uploadThroughput=31250,
            connectionType="wifi")

def outputstart(**kwargs):
    print("START ", kwargs)

tab.call_method("Network.enable", _timeout=20)
tab.set_listener("Network.requestWillBeSent", outputstart)

score 0 · Answer 2 · answered Jul 13 '21 at 13:19

https://stackoverflow.com/a/68363046/8491363

I used selenium (current release 3.0) to read the network log. Check out the link above.

One other way to do it is using Selenium 4.0 but it is still in beta state. When 4.0 gets released, it will officially support Chrome dev tool stuffs.

How to read from the network tab in Python?

2 Answers2