16

I have a directory structure like the following in my serverless application(simplest app to avoid clutter) which I created using AWS SAM with Python 3.8 as the runtime:

├── common
│   └── a.py
├── hello_world
│   ├── __init__.py
│   ├── app.py
│   └── requirements.txt
└── template.yaml

I would like to import common/a.py module inside the Lambda handler - hello_world/app.py. Now I know that I can import it normally in Python by adding the path to PYTHONPATH or sys.path, but it doesn't work when the code is run in Lambda inside a Docker container. When invoked, the Lambda handler function is run inside /var/task directory and the folder structure is not regarded.

I tried inserting /var/task/common, /var/common, /var and even /common to sys.path programmatically like this:

import sys
sys.path.insert(0, '/var/task/common')
from common.a import Alpha

but I still get the error:

ModuleNotFoundError: No module named 'common'

I am aware of Lambda Layers but given my scenario, I would like to directly reference the common code in multiple Lambda functions without the need of uploading to a layer. I want to have something like the serverless-python-requirements plugin in the Serverless framework but in AWS SAM.

So my question is, how should I add this path to common to PYTHONPATH or sys.path? Or is there an alternative workaround for this like [serverless-python-requirements][3] to directly import a module in a parent folder without using Lambda Layers?

Syed Waqas
  • 2,576
  • 4
  • 29
  • 36
  • you have to create package "common" and import it. https://www.geeksforgeeks.org/create-access-python-package/ – Avinash Dalvi Jan 16 '20 at 06:43
  • @aviboy2006 : You mean adding `__init__.py`? Python 3.3+ has implicit namespace packages, which means that there is no requirement anymore for adding `__init__.py` https://stackoverflow.com/questions/37139786/is-init-py-not-required-for-packages-in-python-3-3 As I mentioned, I am using Python 3.8. I also mentioned that I am able to import packages like this in non-lambda environment, I am asking specifically for AWS Lambda(run in a container) which is different from the normal Python execution scenario(`/var/task` is the execution directory when run in AWS Lambda) – Syed Waqas Jan 16 '20 at 07:03
  • 1
    lambda works on virtual environment. /var/task/ create runtime that we will not able to get import path. Better you have create package name "common" folder name. – Avinash Dalvi Jan 16 '20 at 07:04
  • 2
    I am not able to understand this: "Better you have create package name "common" folder name." Do you mean deploying the package separately because like I mentioned the common folder is already a package in Python 3.3+ and I know that Lambda is a virtual environment, that is why I want a workaround. Can you please be more elaborate about this? – Syed Waqas Jan 16 '20 at 07:13
  • 1
    i am saying you cant use common name as folder or import. because it cant override you custom folder to virtual environment folder. you can use "shared" or something different which not comes with system keyword. – Avinash Dalvi Jan 16 '20 at 07:22
  • Renamed `common` to `alphaFolder`, inserted `/alphaFolder`, `/var/alphaFolder` & `/var/task/alphaFolder` to `sys.path` in separate runs; still fails! – Syed Waqas Jan 16 '20 at 07:37
  • Why want to add sys.path? – Avinash Dalvi Jan 16 '20 at 07:39
  • 3
    `sys.path` or no `sys.path`, it doesn't matter to me whatsoever. I just want a way to import modules from the parent folder in a Lambda function handler. If you know the answer, please consider posting one and mention the details! – Syed Waqas Jan 16 '20 at 07:54
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/206047/discussion-between-aviboy2006-and-syed-waqas). – Avinash Dalvi Jan 16 '20 at 08:50
  • @SyedWaqas Since you haven't shared the `template.yaml`, I am not sure what you have under `CodeUri`. I am guessing that you have it as `CodeUri: hello_world/` and that's why it isn't working because SAM would only copy the code inside the `hello_world` directory. To also include the `common` folder, you would need to change your `CodeUri` to `CodeUri: .` and then you would be able to import `from common.a import Alpha`. To verify that your directory structure is correct or the file is even present, add this to the top of your lambda handler `import os` `os.system("ls /path/you/want/")` – Samkit Jain Jul 06 '20 at 17:32
  • If I am correct, you cannot include multiple folders in your `CodeUri`. If your case is like, you have multiple directories at root but only want a couple in the lambda, you won't be able to do that. There are workarounds though and you can have a preprocessing step that updates your directory structure before `sam build`. For example, copying `common/` directory inside the `hello_world/` one. – Samkit Jain Jul 06 '20 at 17:38
  • how did you solve this issue, I am also facing same issue and stuck here,Please share your solution if you can – Md. Parvez Alam Sep 17 '20 at 13:42
  • @Md.ParvezAlam I have shared my solution below. – Syed Waqas Sep 18 '20 at 14:57

2 Answers2

2

I didn't find what I was looking for but I ended up with a solution to create a single Lambda function in the root which handles all the different API calls within the function. Yes my Lambda function is integrated with API Gateway, and I can get the API method and API path using event["httpMethod"] and event ["httpPath"] respectively. I can then put all the packages under the root and import them between each other.

For example, say I have 2 API paths /items and /employees that need to be handled and both of them need to handle both GET and POST methods, the following code suffices:

if event["path"] == '/items':
   if event["httpMethod"] == 'GET':
      ...
   elif event["httpMethod"] == 'POST':
      ...
elif event["path"] == '/employees':
   if event["httpMethod"] == 'GET':
      ...
   if event["httpMethod"] == 'POST':
      ...

So now I can have as much packages under this Lambda function. For example, the following is how repository looks like now:

├── application
│   └── *.py
├── persistence
│   └── *.py
├── models
│   └── *.py
└── rootLambdaFunction.py
└── requirements.txt

This way, I can import packages at will wherever I want within the given structure.

Syed Waqas
  • 2,576
  • 4
  • 29
  • 36
1

I was having a similar issue with package dependencies and according to AWS Knowledge Center you should put all of your items at the root level:

You might need to create a deployment package that includes the function modules located in the root of the .zip file with read and execute permissions for all files.

According to https://aws.amazon.com/premiumsupport/knowledge-center/build-python-lambda-deployment-package/

weez
  • 21
  • 2