Prerequisites Python layer compatibility with Amazon Linux Layer paths for Python runtimes Packaging the layer content Creating the layer Adding the layer to your function Working with manylinux wheel distributions

Working with layers for Python Lambda functions

A Lambda layer is a .zip file archive that contains supplementary code or data. Layers usually contain library dependencies, a custom runtime, or configuration files. Creating a layer involves three general steps:

Package your layer content. This means creating a .zip file archive that contains the dependencies you want to use in your functions.
Create the layer in Lambda.
Add the layer to your functions.

This topic contains steps and guidance on how to properly package and create a Python Lambda layer with external library dependencies.

Topics

Prerequisites
Python layer compatibility with Amazon Linux
Layer paths for Python runtimes
Packaging the layer content
Creating the layer
Adding the layer to your function
Working with manylinux wheel distributions

Prerequisites

To follow the steps in this section, you must have the following:

Python 3.11 and the pip package installer
Amazon Command Line Interface (Amazon CLI) version 2

Throughout this topic, we reference the layer-python sample application on the awsdocs GitHub repository. This application contains scripts that download the dependencies and generate the layers. The application also contains corresponding functions that use dependencies from the layers. After creating a layer, you can deploy and invoke the corresponding function to verify that everything works properly. Because you use the Python 3.11 runtime for the functions, the layers must also be compatible with Python 3.11.

In the layer-python sample application, there are two examples:

The first example involves packaging the requests library into a Lambda layer. The layer/ directory contains the scripts to generate the layer. The function/ directory contains a sample function to help test that the layer works. The majority of this tutorial walks through how to create and package this layer.
The second example involves packaging the numpy library into a Lambda layer. The layer-numpy/ directory contains the scripts to generate the layer. The function-numpy/ directory contains a sample function to help test that the layer works. For an example of how to create and package this layer, see Working with manylinux wheel distributions.

Python layer compatibility with Amazon Linux

The first step to creating a layer is to bundle all of your layer content into a .zip file archive. Because Lambda functions run on Amazon Linux, your layer content must be able to compile and build in a Linux environment.

In Python, most packages are available as wheels (.whl files) in addition to the source distribution. Each wheel is a type of built distribution that supports a specific combination of Python versions, operating systems, and machine instruction sets.

Wheels are useful for ensuring that your layer is compatible with Amazon Linux. When you download your dependencies, download the universal wheel if possible. (By default, pip installs the universal wheel if one is available.) The universal wheel contains any as the platform tag, indicating that it's compatible with all platforms, including Amazon Linux.

In the example that follows, you package the requests library into a Lambda layer. The requests library is an example of a package that's available as a universal wheel.

Not all Python packages are distributed as universal wheels. For example, numpy has multiple wheel distributions, each supporting a different set of platforms. For such packages, download the manylinux distribution to ensure compatibility with Amazon Linux. For detailed instructions about how to package such layers, see Working with manylinux wheel distributions.

In rare cases, a Python package might not be available as a wheel. If only the source distribution (sdist) exists, then we recommend installing and packaging your dependencies in a Docker environment based on the Amazon Linux 2023 base container image. We also recommend this approach if you want to include your own custom libraries written in other languages such as C/C++. This approach mimics the Lambda execution environment in Docker, and ensures that your non-Python package dependencies are compatible with Amazon Linux.

Layer paths for Python runtimes

When you add a layer to a function, Lambda loads the layer content into the /opt directory of that execution environment. For each Lambda runtime, the PATH variable already includes specific folder paths within the /opt directory. To ensure that the PATH variable picks up your layer content, your layer .zip file should have its dependencies in the following folder paths:

python
python/lib/python3.x/site-packages

For example, the resulting layer .zip file that you create in this tutorial has the following directory structure:


layer_content.zip
└ python
    └ lib
        └ python3.11
            └ site-packages
                └ requests
                └ <other_dependencies> (i.e. dependencies of the requests package)
                └ ...

The requests library is correctly located in the python/lib/python3.11/site-packages directory. This ensures that Lambda can locate the library during function invocations.

Packaging the layer content

In this example, you package the Python requests library in a layer .zip file. Complete the following steps to install and package the layer content.

To install and package your layer content

Clone the aws-lambda-developer-guide GitHub repo, which contains the sample code that you need in the sample-apps/layer-python directory.
```
git clone https://github.com/awsdocs/aws-lambda-developer-guide.git
```
Navigate to the layer directory of the layer-python sample app. This directory contains the scripts that you use to create and package the layer properly.
```
cd aws-lambda-developer-guide/sample-apps/layer-python/layer
```
Examine the requirements.txt file. This file defines the dependencies that you want to include in the layer, namely the requests library. You can update this file to include any dependencies that you want to include in your own layer.
Example requirements.txt
```
requests==2.31.0
```
Ensure that you have permissions to run both scripts.
```
chmod 744 1-install.sh && chmod 744 2-package.sh
```
Run the 1-install.sh script using the following command:
```
./1-install.sh
```
This script uses venv to create a Python virtual environment named create_layer. It then installs all required dependencies in the create_layer/lib/python3.11/site-packages directory.
Example 1-install.sh
```
python3.11 -m venv create_layer
source create_layer/bin/activate
pip install -r requirements.txt
```
Run the 2-package.sh script using the following command:
```
./2-package.sh
```
This script copies the contents from the create_layer/lib directory into a new directory named python. It then zips the contents of the python directory into a file named layer_content.zip. This is the .zip file for your layer. You can unzip the file and verify that it contains the correct file structure, as shown in the Layer paths for Python runtimes section.
Example 2-package.sh
```
mkdir python
cp -r create_layer/lib python/
zip -r layer_content.zip python
```

Creating the layer

In this section, you take the layer_content.zip file that you generated in the previous section and upload it as a Lambda layer. You can upload a layer using the Amazon Web Services Management Console or the Lambda API via the Amazon Command Line Interface (Amazon CLI). When you upload your layer .zip file, in the following PublishLayerVersion Amazon CLI command, specify python3.11 as the compatible runtime and arm64 as the compatible architecture.


aws lambda publish-layer-version --layer-name python-requests-layer \
    --zip-file fileb://layer_content.zip \
    --compatible-runtimes python3.11 \
    --compatible-architectures "arm64"

From the response, note the LayerVersionArn, which looks like arn:aws:lambda:us-east-1:123456789012:layer:python-requests-layer:1. You'll need this Amazon Resource Name (ARN) in the next step of this tutorial, when you add the layer to your function.

Adding the layer to your function

In this section, you deploy a sample Lambda function that uses the requests library in its function code, then you attach the layer. To deploy the function, you need a Lambda execution role. If you don't have an existing execution role, follow the steps in the collapsible section. Otherwise, skip to the next section to deploy the function.

To create an execution role

Open the roles page in the IAM console.
Choose Create role.
Create a role with the following properties.
- Trusted entity – Lambda.
- Permissions – AWSLambdaBasicExecutionRole.
- Role name – lambda-role.
The AWSLambdaBasicExecutionRole policy has the permissions that the function needs to write logs to CloudWatch Logs.

To deploy the Lambda function

Navigate to the function/ directory. If you're currently in the layer/ directory, then run the following command:
```
cd ../function
```

Review the function code. The function imports the requests library, makes a simple HTTP GET request, and then returns the status code and body.


import requests

def lambda_handler(event, context):
    print(f"Version of requests library: {requests.__version__}")
    request = requests.get('https://api.github.com/')
    return {
        'statusCode': request.status_code,
        'body': request.text
    }

Create a .zip file deployment package using the following command:
```
zip my_deployment_package.zip lambda_function.py
```

Deploy the function. In the following Amazon CLI command, replace the --role parameter with your execution role ARN:


aws lambda create-function --function-name python_function_with_layer \
    --runtime python3.11 \
    --architectures "arm64" \
    --handler lambda_function.lambda_handler \
    --role arn:aws:iam::123456789012:role/lambda-role \
    --zip-file fileb://my_deployment_package.zip

At this point, you can optionally try to invoke your function before attaching the layer. If you try this, then you should get an import error because your function cannot reference the requests package. To invoke your function, use the following Amazon CLI command:


aws lambda invoke --function-name python_function_with_layer \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' response.json

You should see output that looks like this:


{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

To view the specific error, open the output response.json file. You should see an ImportModuleError with the following error message:


"errorMessage": "Unable to import module 'lambda_function': No module named 'requests'"

Next, attach the layer to your function. In the following Amazon CLI command, replace the --layers parameter with the layer version ARN that you noted earlier:


aws lambda update-function-configuration --function-name python_function_with_layer \
    --cli-binary-format raw-in-base64-out \
    --layers "arn:aws:lambda:us-east-1:123456789012:layer:python-requests-layer:1"

Finally, try to invoke your function using the following Amazon CLI command:


aws lambda invoke --function-name python_function_with_layer \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' response.json

You should see output that looks like this:


{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

The output response.json file contains details about the response.

You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting Amazon resources that you're no longer using, you prevent unnecessary charges to your Amazon Web Services account.

To delete the Lambda layer

Open the Layers page of the Lambda console.
Select the layer that you created.
Choose Delete, then choose Delete again.

To delete the Lambda function

Open the Functions page of the Lambda console.
Select the function that you created.
Choose Actions, Delete.
Type delete in the text input field and choose Delete.

Working with `manylinux` wheel distributions

Sometimes, a package that you want to include as a dependency won't have a universal wheel (specifically, it doesn't have any as the platform tag). In this case, download the wheel that supports manylinux instead. This ensures that your layer libraries are compatible with Amazon Linux.

numpy is one package that doesn't have a universal wheel. If you want to include the numpy package in your layer, then you can complete the following example steps to install and package your layer properly.

To install and package your layer content

Clone the aws-lambda-developer-guide GitHub repo, which contains the sample code that you need in the sample-apps/layer-python directory.
```
git clone https://github.com/awsdocs/aws-lambda-developer-guide.git
```
Navigate to the layer-numpy directory of the layer-python sample app. This directory contains the scripts that you use to create and package the layer properly.
```
cd aws-lambda-developer-guide/sample-apps/layer-python/layer-numpy
```
Examine the requirements.txt file. This file defines the dependencies that you want to include in your layer, namely the numpy library. Here, you specify the URL of the manylinux wheel distribution that's compatible with Python 3.11, Amazon Linux, and the x86_64 instruction set:
Example requirements.txt
```
https://files.pythonhosted.org/packages/3a/d0/edc009c27b406c4f9cbc79274d6e46d634d139075492ad055e3d68445925/numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
```
Ensure that you have permissions to run both scripts.
```
chmod 744 1-install.sh && chmod 744 2-package.sh
```
Run the 1-install.sh script using the following command:
```
./1-install.sh
```
This script uses venv to create a Python virtual environment named create_layer. It then installs all required dependencies in the create_layer/lib/python3.11/site-packages directory. The pip command is different in this case, because you must specify the --platform tag as manylinux2014_x86_64. This tells pip to install the correct manylinux wheel, even if your local machine uses macOS or Windows.
Example 1-install.sh
```
python3.11 -m venv create_layer
source create_layer/bin/activate
pip install -r requirements.txt --platform=manylinux2014_x86_64 --only-binary=:all: --target ./create_layer/lib/python3.11/site-packages
```
Run the 2-package.sh script using the following command:
```
./2-package.sh
```
This script copies the contents from the create_layer/lib directory into a new directory named python. It then zips the contents of the python directory into a file named layer_content.zip. This is the .zip file for your layer. You can unzip the file and verify that it contains the correct file structure as shown in the Layer paths for Python runtimes section.
Example 2-package.sh
```
mkdir python
cp -r create_layer/lib python/
zip -r layer_content.zip python
```

To upload this layer to Lambda, use the following PublishLayerVersion Amazon CLI command:


aws lambda publish-layer-version --layer-name python-numpy-layer \
    --zip-file fileb://layer_content.zip \
    --compatible-runtimes python3.11 \
    --compatible-architectures "x86_64"

From the response, note the LayerVersionArn, which looks like arn:aws:lambda:us-east-1:123456789012:layer:python-numpy-layer:1. To verify that your layer works as expected, deploy the Lambda function in the function-numpy directory.

To deploy the Lambda function

Navigate to the function-numpy/ directory. If you're currently in the layer-numpy/ directory, then run the following command:
```
cd ../function-numpy
```

Review the function code. The function imports the numpy library, creates a simple numpy array, and then returns a dummy status code and body.


import json
import numpy as np

def lambda_handler(event, context):

    x = np.arange(15, dtype=np.int64).reshape(3, 5)
    print(x)

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Create a .zip file deployment package using the following command:
```
zip my_deployment_package.zip lambda_function.py
```

Deploy the function. In the following Amazon CLI command, replace the --role parameter with your execution role ARN:


aws lambda create-function --function-name python_function_with_numpy \
    --runtime python3.11 \
    --handler lambda_function.lambda_handler \
    --role arn:aws:iam::123456789012:role/lambda-role \
    --zip-file fileb://my_deployment_package.zip

Optionally, you can try to invoke your function before attaching the layer. If you try this, then you should get an import error because your function cannot reference the numpy package. To invoke your function, use the following Amazon CLI command:


aws lambda invoke --function-name python_function_with_numpy \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' response.json

You should see output that looks like this:


{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST"
}

To view the specific error, open the output response.json file. You should see an ImportModuleError with the following error message:


"errorMessage": "Unable to import module 'lambda_function': No module named 'numpy'"

Next, attach the layer to your function. In the following Amazon CLI command, replace the --layers parameter with your layer version ARN:


aws lambda update-function-configuration --function-name python_function_with_numpy \
    --cli-binary-format raw-in-base64-out \
    --layers "arn:aws:lambda:us-east-1:123456789012:layer:python-requests-layer:1"

Finally, try to invoke your function using the following Amazon CLI command:


aws lambda invoke --function-name python_function_with_numpy \
    --cli-binary-format raw-in-base64-out \
    --payload '{ "key": "value" }' response.json

You should see output that looks like this:


{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

You can examine the function logs to verify that the code prints the numpy array to standard out.

Javascript is disabled or is unavailable in your browser.

To use the Amazon Web Services Documentation, Javascript must be enabled. Please refer to your browser's Help pages for instructions.

Deploy container images

Context

Working with layers for Python Lambda functions

Topics

Prerequisites

Python layer compatibility with Amazon Linux

Layer paths for Python runtimes

Packaging the layer content

To install and package your layer content

Example requirements.txt

Example 1-install.sh

Example 2-package.sh

Creating the layer

Adding the layer to your function

To create an execution role

To deploy the Lambda function

To delete the Lambda layer

To delete the Lambda function

Working with manylinux wheel distributions

To install and package your layer content

Example requirements.txt

Example 1-install.sh

Example 2-package.sh

To deploy the Lambda function

Working with `manylinux` wheel distributions