Pyppeteer: Browser closed unexpectedly in AWS Lambda

Question

I'm running into this error in AWS Lambda. It appears that the devtools websocket is not up. Not sure how to fix it. Any ideas? Thanks for your time.

Exception originated from get_ws_endpoint() due to websocket response timeout https://github.com/pyppeteer/pyppeteer/blob/ad3a0a7da221a04425cbf0cc92e50e93883b077b/pyppeteer/launcher.py#L225

Lambda code:

import os
import json
import asyncio
import logging
import boto3
import pyppeteer
from pyppeteer import launch

logger = logging.getLogger()
logger.setLevel(logging.INFO)

pyppeteer.DEBUG = True  # print suppressed errors as error log

def lambda_handler(event, context):
    asyncio.get_event_loop().run_until_complete(main())

async def main():
    browser = await launch({
        'headless': True,
        'args': [
            '--no-sandbox'
            ]
    })
    page = await browser.newPage()
    await page.goto('http://example.com')
    await page.screenshot({'path': '/tmp/example.png'})
    await browser.close()
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Exception:

    Response:
{
  "errorMessage": "Browser closed unexpectedly:\n",
  "errorType": "BrowserError",
  "stackTrace": [
    "  File \"/var/task/lambda_handler.py\", line 23, in lambda_handler\n    asyncio.get_event_loop().run_until_complete(main())\n",
    "  File \"/var/lang/lib/python3.8/asyncio/base_events.py\", line 616, in run_until_complete\n    return future.result()\n",
    "  File \"/var/task/lambda_handler.py\", line 72, in main\n    browser = await launch({\n",
    "  File \"/opt/python/pyppeteer/launcher.py\", line 307, in launch\n    return await Launcher(options, **kwargs).launch()\n",
    "  File \"/opt/python/pyppeteer/launcher.py\", line 168, in launch\n    self.browserWSEndpoint = get_ws_endpoint(self.url)\n",
    "  File \"/opt/python/pyppeteer/launcher.py\", line 227, in get_ws_endpoint\n    raise BrowserError('Browser closed unexpectedly:\\n')\n"
  ]
}

Request ID:
"06be0620-8b5c-4600-a76e-bc785210244e"

Function Logs:
START RequestId: 06be0620-8b5c-4600-a76e-bc785210244e Version: $LATEST
---- files in /tmp ----
[W:pyppeteer.chromium_downloader] start chromium download.
Download may take a few minutes.

  0%|          | 0/108773488 [00:00<?, ?it/s]
 11%|█▏        | 12267520/108773488 [00:00<00:00, 122665958.31it/s]
 27%|██▋       | 29470720/108773488 [00:00<00:00, 134220418.14it/s]
 42%|████▏     | 46172160/108773488 [00:00<00:00, 142570388.86it/s]
 58%|█████▊    | 62607360/108773488 [00:00<00:00, 148471487.93it/s]
 73%|███████▎  | 79626240/108773488 [00:00<00:00, 154371569.93it/s]
 88%|████████▊ | 95754240/108773488 [00:00<00:00, 156353972.12it/s]
100%|██████████| 108773488/108773488 [00:00<00:00, 161750092.47it/s]
[W:pyppeteer.chromium_downloader] 
chromium download done.
[W:pyppeteer.chromium_downloader] chromium extracted to: /tmp/local-chromium/588429
-----
/tmp/local-chromium/588429/chrome-linux/chrome
[ERROR] BrowserError: Browser closed unexpectedly:

Traceback (most recent call last):
  File "/var/task/lambda_handler.py", line 23, in lambda_handler
    asyncio.get_event_loop().run_until_complete(main())
  File "/var/lang/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/var/task/lambda_handler.py", line 72, in main
    browser = await launch({
  File "/opt/python/pyppeteer/launcher.py", line 307, in launch
    return await Launcher(options, **kwargs).launch()
  File "/opt/python/pyppeteer/launcher.py", line 168, in launch
    self.browserWSEndpoint = get_ws_endpoint(self.url)
  File "/opt/python/pyppeteer/launcher.py", line 227, in get_ws_endpoint
    raise BrowserError('Browser closed unexpectedly:\n')END RequestId: 06be0620-8b5c-4600-a76e-bc785210244e
REPORT RequestId: 06be0620-8b5c-4600-a76e-bc785210244e  Duration: 33370.61 ms   Billed Duration: 33400 ms   Memory Size: 3008 MB    Max Memory Used: 481 MB Init Duration: 445.58 ms

This is unrelated to AWS Lambda, but since this question is the top Google hit for "BrowserError: Browser closed unexpectedly"... Using a proxy? See https://stackoverflow.com/a/69233003/880783 — bers, Sep 18 '21 at 08:39

score 19 · Answer 1 · answered Jun 10 '20 at 14:04

I think BrowserError: Browser closed unexpectedly is just the error you get when Chrome crashes for whatever reason. It would be nice if pyppeteer printed out the error, but it doesn't.

To track things down, it's helpful to pull up the exact command that pyppeteer runs. You can do that this way:

>>> from pyppeteer.launcher import Launcher
>>> ' '.join(Launcher().cmd)
/root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome --disable-background-networking --disable-background-timer-throttling --disable-breakpad --disable-browser-side-navigation --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=site-per-process --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-translate --metrics-recording-only --no-first-run --safebrowsing-disable-auto-update --enable-automation --password-store=basic --use-mock-keychain --headless --hide-scrollbars --mute-audio about:blank --no-sandbox --remote-debugging-port=33423 --user-data-dir=/root/.local/share/pyppeteer/.dev_profile/tmp5cj60q6q

When I ran that command in my Docker image, I got the following error:

$ /root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome # ...
/root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome:
error while loading shared libraries:
libnss3.so: cannot open shared object file: No such file or directory

So I installed libnss3:

apt-get install -y libnss3

Then I ran the command again and got a different error:

$ /root/.local/share/pyppeteer/local-chromium/588429/chrome-linux/chrome # ...
[0609/190651.188666:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

So I needed to change my launch command to something like:

browser = await launch(headless=True, args=['--no-sandbox'])

and now it works!

Thank you for the tip. How would I fix the issue with libnss3 on AWS lambda? There is no way to install low level deps directly on lambda. — Sudhakar, Jun 10 '20 at 18:24

score 6 · Accepted Answer · answered May 14 '20 at 01:03

Answering my own question.

Finally I was able to run Pyppeteer(v0.2.2) with Python 3.6 and 3.7 (not 3.8) after I bundled chromium binary in a lambda layer.

So in summary, it appears to work only when its configured to run with user provided chromium executable path and not with automatically downloaded chrome. Probably some race condition or something.

Got Chromium from https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-41/stable-headless-chromium-amazonlinux-2017-03.zip

browser = await launch(
        headless=True, 
        executablePath='/opt/python/headless-chromium',
        args=[
            '--no-sandbox',
            '--single-process',
            '--disable-dev-shm-usage',
            '--disable-gpu',
            '--no-zygote'
        ])

Issue posted on repo https://github.com/pyppeteer/pyppeteer/issues/108

I logged in specifically to upvote this - lifesaving answer. For those who may not know, the chromium binary should be inside a folder "python" (not /opt/python) , zip python folder and create the layer. For testing, you can also bundle the binary with your code and use path as ./headless-chromium. Although, layers approach is what you should be going after. — Ian McGrath, Oct 16 '21 at 07:48

score 2 · Answer 3 · answered Jul 24 '20 at 15:06

I have been trying to run pyppeteer in a Docker container and ran into the same issue.

Finally managed to fix it thanks to this comment: https://github.com/miyakogi/pyppeteer/issues/14#issuecomment-348825238

I installed Chrome manually through apt

curl -sSL https://dl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google-chrome.list
apt update -y && apt install -y google-chrome-stable

and then specified the path when launching the browser. You also have to run it in headless and with args "--no-sandbox"

browser = await launch(executablePath='/usr/bin/google-chrome-stable', headless=True, args=['--no-sandbox'])

Hope this will help!

score 1 · Answer 4 · answered Jan 20 '21 at 16:10

1

If anybody is running on Heroku and facing the same error:

Add the buildpack : The url for the buildpack is below :

https://github.com/jontewks/puppeteer-heroku-buildpack

Ensure that you're using --no-sandbox mode

launch({ args: ['--no-sandbox'] })

answered Jan 20 '21 at 16:10

aahnik

1,661
1
11
29

score 1 · Answer 5 · answered Mar 08 '22 at 08:09

Make sure all the necessary dependencies are installed. You can run ldd /path/to/your/chrome | grep not on a Linux machine to check which dependencies are missing.

In my case, i get this:

libatk-bridge-2.0.so.0 => not found
libgtk-3.so.0 => not found

and then install dependencies:

sudo apt-get install at-spi2-atk gtk3

and now it works!

Pyppeteer: Browser closed unexpectedly in AWS Lambda

5 Answers5

Linked