Unable to import module 'lambda_function': No module named 'pandas'

Question

START RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99 Version: $LATEST Unable to import module 'lambda_function': No module named 'pandas'
END RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99

I'm using Windows 7 64-bit as the host OS.

What I want to do

I simply want to use pandas in AWS-Lambda environment. Just like I use it in windows environment, I am looking for a simple solution for Lambda.

What I have tried so far

• Installed Xubuntu on a virtual box.
• Create a virtual environment called myvenv in Xubuntu on virtual-box.
• Then I installed pandas3.6 in myvenv.
• Thereafter, I copied the folder contents in myvenv at location '/usr/local/lib/python3.6/site-packages/' to my host OS.
• In the host OS (windows 7), I created a folder called packs, pasted the contents of myvenv.
• created a lambda_function.py script in packs in host OS (windows 7)
• I then zipped the folder packs using 7zip software and upload it as zip in Lambda
• In Lambda, the lambda function handler name is, lambda_handler(). The code snippet looks like,

import pandas as pd

def lambda_handler(event, context):

    dates = pd.date_range('2019001', periods=6)

    df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))
    print(df)

• The handler is named as lambda_function.lambda_handler. I have given the lambda-role AWSLambdaFullAccess permission.
• The time out is set to 4 min and 3 sec.

• The test event looks like

  { "key1": "This will be printed if all OK" }

I have tried the following solutions:

• Tried precompiled linux-compatible binaries for pandas & numpy from here -- no luck.
• In Lambda, changed the Handler info to python_filename.function_name. For my case, it was lambda_function.lambda_handler -- failed with no module named 'pandas' error.
• placed the lambda function in the root folder, zipped the folder using 7zip software and upload the folder to the S3 bucket. For my case, I placed the function at location python\lib\python3.6\site_packages\lambda_function.pyfailed with no module named 'pandas' error.
• Already tried these related solutions posted on SO, 1, 2, 3, 4, 5, 6

Note: I do not want to use Docker, because I do not know how to use it and I'm not willing to learn it as I'm exasperated now. I'm coming from a windows environment (it sucks, I now know.)

Any ideas on how to get this to work.

Answer 1

I was able to import the pandas library successfully using a Lambda layer and an Amazon linux Cloud 9 instance. There are the commands I executed in the Cloud 9 instance and the Lambda function's output. I had to change the code slightly since it was failing with an import error and string value error.

Alternatively, these commands can also be executed in an EC2 instance. If it's not possible to use the SAM CLI(which uses docker) or just plain docker on windows we'll need to use an Amazon Linux instance to build everything since that's what AWS Lambda uses currently. I don't believe using an ubuntu instance will work here.

Commands:

python --version
Python 3.6.8

# https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
# python 3.6 uses Amazon Linux currently 

mkdir project
cd project
virtualenv v-env
source ./v-env/bin/activate
pip install pandas
deactivate

# creating layer
# https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path
mkdir python
cd python
cp -r ../v-env/lib64/python3.6/dist-packages/* .
cd ..
zip -r panda_layer.zip python
aws lambda publish-layer-version --layer-name pandas --zip-file fileb://panda_layer.zip --compatible-runtimes python3.6 

The publish-later-version command will create a new AWS Lambda layer in the region given in the command or the config file for the CLI.

A Lambda layer will apply the library to the Lambda function's code without needing to apply it directly to the deployment package. This also allows the use of the online code editor in Lambda since the deployment package is under the limit of 3MB

I applied the Lambda layer by clicking on the Layer button in the web console and choosing the layer version that I most recently published. I have a second version there because the first time I attempted this is put the contents of the lib directory which isn't for a 64 bit OS and my code failed in AWS Lambda.

Alternatively, you can also apply the layer using the CLI command update-function-configuration

Lambda function code I used:

import pandas as pd
import numpy as np

def lambda_handler(event, context):
    dates = pd.date_range(start='1/1/2018', end='1/08/2018')
    df = pd.DataFrame(np.random.randn(8, 4), index=dates, columns=list('ABCD'))
    print(str(df))

---

Lambda output:

START RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850 Version: $LATEST
                   A         B         C         D
2018-01-01 -1.040318  0.450841 -0.381687 -0.105480
2018-01-02 -1.381793 -0.481572  0.828419 -0.885205
2018-01-03  1.437799 -0.649816 -0.577112  0.400670
2018-01-04 -0.730997 -0.778775 -1.514203  1.165661
2018-01-05  1.963595 -1.137054  0.920218  0.960210
2018-01-06 -0.429179 -0.745549  1.482562  0.298623
2018-01-07 -1.082388 -0.529476 -1.051663  1.616683
2018-01-08  0.042779 -2.338471 -0.142992  0.680399
END RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850
REPORT RequestId: 27f09b6c-a4cd-49dd-bb3e-ae6fc7cd5850  Duration: 536.76 ms Billed Duration: 600 ms Memory Size: 128 MB Max Memory Used: 122 MB Init Duration: 1721.51 ms   
XRAY TraceId: 1-5d741e40-1311daa29fc16c74735988fc   SegmentId: 61a595dd3492c331 Sampled: false  

Answer 2

To use pandas in an AWS Lambda environment with a Python runtime, the simplest way is to:

• Scroll down in Code tab of Lambda console
• "Add Layer"
• Choose "AWS Layers" (choose a layer from a list of layers provided by AWS)
• AWSDataWrangler-Python37 (which includes pandas)

If you're working with automation for the deployment, you'll want to find the ARN. Even in the console, you'll probably want to just take the extra step and choose the "Specify an ARN" option, since it seems like the dropdown menu for "AWS layers" isn't caught up with the latest AWS Layers that are actually available.

You can find a library of all the AWS-hosted Lambda layers at https://serverlessrepo.aws.amazon.com/applications. If you search for pandas here, you'll find 15 results at the moment, including aws-sdk-pandas-layer-py3-7 (3.8 and 3.9 are there too). If you click into the detail page for that layer, you'll see the arn for that AWS-hosted layer.

You can also just click "Deploy" on that details page. Then the layer will be available to you from the "Custom layers" dropdown in your Lambda console.

Answer 3

I found this github repo that has pre-built package ARNs. Find the one you want for your AWS region, and then choose "Specify an ARN" when creating the layer and paste in the layer ARN from this github repo:

https://github.com/keithrozario/Klayers

Answer 4

For the layer approach, make sure that the contents of the uploaded layer package (the site-packages generated by virtualenv) are contained inside a folder/directory named python. Like after you unzip the package it should create a python named directory and have contents of site-packages.

 cd venv/lib/python3.6
 mkdir python
 cp -r site-packages/* python
 zip layer.zip python

For the approach with the dependencies in the same zip, this structure will be different.