5

I want to use an automated browser and execute my steps with jupyter notebook cells instead of using .py scripts. This works fine with the browser automation library called selenium.

It does not work fine with the library called Playwright. In fact it doesn't work at all. I tried every single line of code they provided in their manual. NOTHING works in jupyter notebooks. EVERYTHING works fine on my machine as long as copy-pasting the same code in some .py file and executing it. Various examples that I'm talking about can be found here: https://playwright.dev/python/docs/intro

I really don't get why I'm unable to make it work in a jupyter notebook, especially if it works fine in literally every .py file.

Edit: Apparently it works on mac but I use windows

Noahela
  • 59
  • 2
  • 5
  • 1
    Sympathetic to the problems, but this description would have benefited from some specific details about what's not working. "NOTHING" isn't that helpful, it doesn't indicate the nsture of any error messages, when they occur or other sensible details to diagnose the situation – Neil Apr 28 '22 at 13:31
  • no it would have not. i mean it like i say it. literally. you take ANY code example from their examples u put it into a .py script and it works as intended. use a jupyter notebook and nothing works anymore. NOTHING. not hard to understand isnt it? – Noahela Jul 27 '22 at 07:13
  • 1
    My point was not about the ease of understanding, it was that the question is not particularly well written from the perspective of others being able to assist you. – Neil Jul 29 '22 at 12:23
  • thank you for your feedback. here the solution that i would want: take ANY code example from the link i provided, make it work in a jupyter notebook (using windows) and problem solved – Noahela Sep 05 '22 at 05:40

4 Answers4

9

The code below works on both MacOS and Linux.

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

enter image description here

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''
Ferris
  • 5,325
  • 1
  • 14
  • 23
  • 1
    This implies there's a fix, but despite the various GitHub issues being formally closed, so far as I'm aware this is still actually not possible. See comments here: https://github.com/microsoft/playwright-python/issues/178#issuecomment-1062232487 – Neil Apr 28 '22 at 13:35
  • In fact, it works in my jupyter notebook. I can test code in it. – Ferris Apr 29 '22 at 01:36
  • 2
    Thanks Ferris for the update. It looks like you're on a Mac which likely explains it but on Windows (and WSL according to the comment) i understand it still isn't working. Am AFK right now but I'll double check your solution again soon – Neil Apr 30 '22 at 07:23
  • 2
    i use windows and this code throws an error for me. but its great if it at least works on mac – Noahela May 31 '22 at 11:59
  • This won't work on Windows. But, see my answer below for a solution that does work (at least for me). – gloomyfit Jul 04 '23 at 15:24
3

Since Colab notebooks are hosted Jupyter Notebooks, I recommend the following solution for running playwright in your hosted Jupyter instance.

I have only tested in my Google Colab notebook and have not tested in a locally hosted Jupyter instance.

Playwright in Google Colab Solution

gohaku
  • 31
  • 1
  • 2
    The OP's question is about running Playwright with [Jupyter on Windows](https://stackoverflow.com/questions/71228742/how-to-use-the-playwright-library-in-jupyter-notebook-instead-of-using-a-py-scr/74877608#comment129978315_71228742). Not sure what Google Colab has to do with OP's question. – Vijay Varadan Dec 23 '22 at 20:48
3

If you're experiencing issues running Playwright inside of a Jupyter notebook on Windows, try disabling the event loop policy of the kernel.

  1. Navigate to Lib/site-packages/ipykernel/kernelapp.py in your Python directory.
  2. Disable the following line: asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())
if sys.platform.startswith("win") and sys.version_info >= (3, 8):
    import asyncio

    try:
        from asyncio import WindowsProactorEventLoopPolicy, WindowsSelectorEventLoopPolicy
    except ImportError:
        pass
        # not affected
    else:
        if type(asyncio.get_event_loop_policy()) is WindowsProactorEventLoopPolicy:
            # WindowsProactorEventLoopPolicy is not compatible with tornado 6
            # fallback to the pre-3.8 default of Selector
            # asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())
            pass
gloomyfit
  • 475
  • 1
  • 5
  • 13
  • 1
    Glad to hear that it works on Windows. Could you provide more related documents/links about the solution? – Ferris Jul 05 '23 at 02:26
  • 1
    I wish, I found this solution close to half a year ago. And, I stumbled upon this question when trying to solve it once again on a different project (I had forgotten what I did the last time around; but found a note mentioning it in another project of mine, and it worked once more so figured I'd post it here). Might've gleaned it from an open github issue or something. – gloomyfit Jul 05 '23 at 11:21
  • super busy rn, cant verify answer... should i mark it as solution? your call @Ferris – Noahela Jul 07 '23 at 13:31
1

If you can't use async API in jupyter notebook, you can try to create a virtual environment for playwright:

In terminal:

# create a virtual environment for playwright
python3 -m venv playwright_new
source ~/playwright_new/bin/activate
pip install playwright ipykernel requests 
playwright install

Then, create a kernel link for jupyter notebook:

source ~/playwright_new/bin/activate

# create kernel link for jupyter notebook 
python -m ipykernel install --user --name playwright_new --display-name "playwright_new"

# in mac
ls /Users/xxx/Library/Jupyter/kernels/
tree /Users/xxx/Library/Jupyter/kernels/playwright_new
    /Users/xxx/Library/Jupyter/kernels/playwright_new
    ├── kernel.json
    ├── logo-32x32.png
    └── logo-64x64.png

# or in linux
tree /root/.local/share/jupyter/kernels

Then, run the python code again.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

Error debug


if exec python code throws an Error:

Error: Executable doesn't exist at /Users/xxxx/Library/Caches/ms-playwright/chromium-1000/chrome-mac/Chromium.app/Contents/MacOS/Chromium
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers:              ║
║                                                                         ║
║     playwright install                                                  ║
║                                                                         ║
║ <3 Playwright Team                                                      ║
╚═════════════════════════════════════════════════════════════════════════╝

In terminal:

# you already install playwright
playwright install 

cd /Users/xxxx/Library/Caches/ms-playwright
ls
    chromium-978106/ ffmpeg-1007/     firefox-1319/    webkit-1616/

# but the folder ms-playwright/chromium-1000 NOT EXISTS
# COPY the exists chromium folder with a new name  `chromium-1000`
cp -r chromium-978106 chromium-1000
Ferris
  • 5,325
  • 1
  • 14
  • 23