Tutorial: Using Amazon Lambda with Amazon DocumentDB Streams - Amazon Lambda
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Tutorial: Using Amazon Lambda with Amazon DocumentDB Streams

In this tutorial, you create a basic Lambda function that consumes events from an Amazon DocumentDB (with MongoDB compatibility) change stream. To complete this tutorial, you will go through the following stages:

  • Set up your Amazon DocumentDB cluster, connect to it, and activate change streams on it.

  • Create your Lambda function, and configure your Amazon DocumentDB cluster as an event source for your function.

  • Test the end-to-end setup by inserting items into your Amazon DocumentDB database.

Prerequisites

If you do not have an Amazon Web Services account, use the following procedure to create one.

To sign up for Amazon Web Services
  1. Open http://www.amazonaws.cn/ and choose Sign Up.

  2. Follow the on-screen instructions.

Amazon sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to http://www.amazonaws.cn/ and choosing My Account.

After you sign up for an Amazon Web Services account, safeguard your administrative user by turning on multi-factor authentication (MFA). For instructions, see Enable a virtual MFA device for an IAM user (console) in the IAM User Guide.

To give other users access to your Amazon Web Services account resources, create IAM users. To secure your IAM users, turn on MFA and only give the IAM users the permissions needed to perform their tasks.

For more information about creating and securing IAM users, see the following topics in the IAM User Guide:

If you have not yet installed the Amazon Command Line Interface, follow the steps at Installing or updating the latest version of the Amazon CLI to install it.

The tutorial requires a command line terminal or shell to run commands. In Linux and macOS, use your preferred shell and package manager.

Note

In Windows, some Bash CLI commands that you commonly use with Lambda (such as zip) are not supported by the operating system's built-in terminals. To get a Windows-integrated version of Ubuntu and Bash, install the Windows Subsystem for Linux.

Create the Amazon Cloud9 environment

Step 1 create a Amazon Cloud9 environment

Before creating the Lambda function, you need to create and configure your Amazon DocumentDB cluster. The steps to set up your cluster in this tutorial is based on the procedure in Get Started with Amazon DocumentDB.

Note

If you already have a Amazon DocumentDB cluster set up, ensure that you activate change streams and create the necessary interface VPC endpoints. Then, you can skip directly to the function creation steps.

First, create an Amazon Cloud9 environment. You’ll use this environment throughout this tutorial to connect to and query your DocumentDB cluster.

To create an Amazon Cloud9 environment
  1. Open the Cloud9 console and choose Create environment.

  2. Create an environment with the following configuration:

    • Under Details:

      • NameDocumentDBCloud9Environment

      • Environment type – New EC2 instance

    • Under New EC2 instance:

      • Instance typet2.micro (1 GiB RAM + 1 vCPU)

      • Platform – Amazon Linux 2

      • Timeout – 30 minutes

    • Under Network settings:

      • Connection – Amazon Systems Manager (SSM)

      • Expand the VPC settings dropdown.

      • Amazon Virtual Private Cloud (VPC) – Choose your default VPC.

      • Subnet – No preference

    • Keep all other default settings.

  3. Choose Create. Provisioning your new Amazon Cloud9 environment can take several minutes.

Create the EC2 security group

Step 2 create an EC2 security group

Next, create a EC2 security group with rules that allow traffic between your DocumentDB cluster and your Cloud9 environment.

To create an EC2 security group
  1. Open the EC2 console. Under Network and Security, choose Security groups.

  2. Choose Create security group.

  3. Create a security group with the following configuration:

    • Under Basic details:

      • Security group nameDocDBTutorial

      • Description – Security group for traffic between Cloud9 and DocumentDB.

      • VPC – Choose your default VPC.

    • Under Inbound rules, choose Add rule. Create a rule with the following configuration:

      • Type – Custom TCP

      • Port range – 27017

      • Source – Custom

      • In the search box next to Source, choose the security group for the Amazon Cloud9 environment you created in the previous step. To see a list of available security groups, enter cloud9 in the search box. Choose the security group with the name aws-cloud9-<environment_name>.

    • Keep all other default settings.

  4. Choose Create security group.

Create the DocumentDB cluster

Step 3 create a DocumentDB cluster

In this step, you’ll create a DocumentDB cluster using the security group from the previous step.

To create a DocumentDB cluster
  1. Open the DocumentDB console. Under Clusters, choose Create.

  2. Create a cluster with the following configuration:

    • For Cluster type, choose Instance Based Cluster.

    • Under Configuration:

      • Engine version – 5.0.0

      • Instance class – db.t3.medium (free trial eligible)

      • Number of instances – 1.

    • Under Authentication:

      • Enter the Username and Password needed to connect to your cluster (same credentials as you used to create the secret in the previous step). In Confirm password, confirm your password.

    • Toggle on Show advanced settings.

    • Under Network settings:

      • Virtual Private Cloud (VPC) – Choose your default VPC.

      • Subnet group – default

      • VPC security groups – In addition to default (VPC), choose the DocDBTutorial (VPC) security group you created in the previous step.

    • Keep all other default settings.

  3. Choose Create cluster. Provisioning your DocumentDB cluster can take several minutes.

Create the secret in Secrets Manager

Step 4 create a secret in Secrets Manager

To access your DocumentDB cluster manually, you must provide username and password credentials. For Lambda to access your cluster, you must provide a Secrets Manager secret that contains these same access credentials when setting up your event source mapping. In this step, you’ll create this secret.

To create the secret in Secrets Manager
  1. Open the Secrets Manager console and choose Store a new secret.

  2. For Choose secret type, choose the following options:

    • Under Basic details:

      • Secret type – Credentials for Amazon DocumentDB database

      • Under Credentials, enter the username and password you’ll use to access your DocumentDB cluster.

      • Database – Choose your DocumentDB cluster.

      • Choose Next.

  3. For Configure secret, choose the following options:

    • Secret nameDocumentDBSecret

    • Choose Next.

  4. Choose Next.

  5. Choose Store.

  6. Refresh the console to verify that you successfully stored the DocumentDBSecret secret.

Note down the Secret ARN of your secret. You’ll need it in a later step.

Install the mongo shell

Step 5 install the mongo shell

In this step, you’ll install the mongo shell in your Cloud9 environment. The mongo shell is a command-line utility that you use to connect to and query your DocumentDB cluster.

To install the mongo shell on your Cloud9 environment
  1. Open the Cloud9 console. Next to the DocumentDBCloud9Environment environment you created earlier, click on the Open link under the Cloud9 IDE column.

  2. In the terminal window, create the MongoDB repository file with the following command:

    echo -e "[mongodb-org-5.0] \nname=MongoDB Repository\nbaseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/5.0/x86_64/\ngpgcheck=1 \nenabled=1 \ngpgkey=https://www.mongodb.org/static/pgp/server-5.0.asc" | sudo tee /etc/yum.repos.d/mongodb-org-5.0.repo
  3. Then, install the mongo shell with the following command:

    sudo yum install -y mongodb-org-shell
  4. To encrypt data in transit, download the public key for Amazon DocumentDB. The following command downloads a file named global-bundle.pem:

    wget https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

Connect to the DocumentDB cluster

Step 6 connect to the DocumentDB cluster

You’re now ready to connect to your DocumentDB cluster using the mongo shell.

To connect to your DocumentDB cluster
  1. Open the DocumentDB console. Under Clusters, choose your cluster by choosing its cluster identifier.

  2. In the Connectivity & security tab, under Connect to this cluster with the mongo shell, choose Copy.

  3. In your Cloud9 environment, paste this command into the terminal. Replace <insertYourPassword> with the correct password.

After entering this command, if the command prompt becomes rs0:PRIMARY>, then you’re connected to your Amazon DocumentDB cluster.

Activate change streams

Step 7 activate change streams

For this tutorial, you’ll track changes to the products collection of the docdbdemo database in your DocumentDB cluster. You do this by activating change streams. First, create the docdbdemo database and test it by inserting a record.

To create a new database within your cluster
  1. In your Cloud9 environment, ensure that you’re still connected to your DocumentDB cluster.

  2. In the terminal window, use the following command to create a new database called docdbdemo:

    use docdbdemo
  3. Then, use the following command to insert a record into docdbdemo:

    db.products.insert({"hello":"world"})

    You should see output that looks like this:

    WriteResult({ "nInserted" : 1 })
  4. Use the following command to list all databases:

    show dbs

    Ensure that your output contains the docdbdemo database:

    docdbdemo 0.000GB

Next, activate change streams on the products collection of the docdbdemo database using the following command:

db.adminCommand({modifyChangeStreams: 1, database: "docdbdemo", collection: "products", enable: true});

You should see output that looks like this:

{ "ok" : 1, "operationTime" : Timestamp(1680126165, 1) }

Create interface VPC endpoints

Step 8 create interface VPC endpoints

Next, create interface VPC endpoints to ensure that Lambda and Secrets Manager (used later to store our cluster access credentials) can connect to your default VPC.

To create interface VPC endpoints
  1. Open the VPC console. In the left menu, under Virtual private cloud, choose Endpoints.

  2. Choose Create endpoint. Create an endpoint with the following configuration:

    • For Name tag, enter lambda-default-vpc.

    • For Service category, choose Amazon services.

    • For Services, enter lambda in the search box. Choose the service with format com.amazonaws.<region>.lambda.

    • For VPC, choose your default VPC.

    • For Subnets, check the boxes next to each availability zone. Choose the correct subnet ID for each availability zone.

    • For IP address type, select IPv4.

    • For Security groups, choose the default VPC security group (Group name of default), and the security group you created earlier (Group name of DocDBTutorial).

    • Keep all other default settings.

    • Choose Create endpoint.

  3. Again, choose Create endpoint. Create an endpoint with the following configuration:

    • For Name tag, enter secretsmanager-default-vpc.

    • For Service category, choose Amazon services.

    • For Services, enter secretsmanager in the search box. Choose the service with format com.amazonaws.<region>.secretsmanager.

    • For VPC, choose your default VPC.

    • For Subnets, check the boxes next to each availability zone. Choose the correct subnet ID for each availability zone.

    • For IP address type, select IPv4.

    • For Security groups, choose the default VPC security group (Group name of default), and the security group you created earlier (Group name of DocDBTutorial).

    • Keep all other default settings.

    • Choose Create endpoint.

This completes the cluster setup portion of this tutorial.

Create the execution role

Step 9 create the execution role

In the next set of steps, you’ll create your Lambda function. First, you need to create the execution role that gives your function permission to access your cluster. You do this by creating an IAM policy first, then attaching this policy to an IAM role.

To create IAM policy
  1. Open the Policies page in the IAM console and choose Create policy.

  2. Choose the JSON tab. In the following policy, replace the Secrets Manager resource ARN in the final line of the statement with your secret ARN from earlier, and copy the policy into the editor.

    { "Version": "2012-10-17", "Statement": [ { "Sid": "LambdaESMNetworkingAccess", "Effect": "Allow", "Action": [ "ec2:CreateNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DescribeVpcs", "ec2:DeleteNetworkInterface", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "kms:Decrypt" ], "Resource": "*" }, { "Sid": "LambdaDocDBESMAccess", "Effect": "Allow", "Action": [ "rds:DescribeDBClusters", "rds:DescribeDBClusterParameters", "rds:DescribeDBSubnetGroups" ], "Resource": "*" }, { "Sid": "LambdaDocDBESMGetSecretValueAccess", "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:DocumentDBSecret" } ] }
  3. Choose Next: Tags, then choose Next: Review.

  4. For Name, enter AWSDocumentDBLambdaPolicy.

  5. Choose Create policy.

To create the IAM role
  1. Open the Roles page in the IAM console and choose Create role.

  2. For Select trusted entity, choose the following options:

    • Trusted entity type – Amazon service

    • Use case – Lambda

    • Choose Next.

  3. For Add permissions, choose the AWSDocumentDBLambdaPolicy policy you just created, as well as the AWSLambdaBasicExecutionRole to give your function permissions to write to Amazon CloudWatch Logs.

  4. Choose Next.

  5. For Role name, enter AWSDocumentDBLambdaExecutionRole.

  6. Choose Create role.

Create the Lambda function

Step 10 create the Lambda function

The following example code receives a DocumentDB event input and processes the message that it contains.

Go
SDK for Go V2
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Go.

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 package main import ( "context" "encoding/json" "fmt" "github.com/aws/aws-lambda-go/lambda" ) type Event struct { Events []Record `json:"events"` } type Record struct { Event struct { OperationType string `json:"operationType"` NS struct { DB string `json:"db"` Coll string `json:"coll"` } `json:"ns"` FullDocument interface{} `json:"fullDocument"` } `json:"event"` } func main() { lambda.Start(handler) } func handler(ctx context.Context, event Event) (string, error) { fmt.Println("Loading function") for _, record := range event.Events { logDocumentDBEvent(record) } return "OK", nil } func logDocumentDBEvent(record Record) { fmt.Printf("Operation type: %s\n", record.Event.OperationType) fmt.Printf("db: %s\n", record.Event.NS.DB) fmt.Printf("collection: %s\n", record.Event.NS.Coll) docBytes, _ := json.MarshalIndent(record.Event.FullDocument, "", " ") fmt.Printf("Full document: %s\n", string(docBytes)) }
JavaScript
SDK for JavaScript (v3)
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using JavaScript.

console.log('Loading function'); exports.handler = async (event, context) => { event.events.forEach(record => { logDocumentDBEvent(record); }); return 'OK'; }; const logDocumentDBEvent = (record) => { console.log('Operation type: ' + record.event.operationType); console.log('db: ' + record.event.ns.db); console.log('collection: ' + record.event.ns.coll); console.log('Full document:', JSON.stringify(record.event.fullDocument, null, 2)); };
Python
SDK for Python (Boto3)
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Python.

import json def lambda_handler(event, context): for record in event.get('events', []): log_document_db_event(record) return 'OK' def log_document_db_event(record): event_data = record.get('event', {}) operation_type = event_data.get('operationType', 'Unknown') db = event_data.get('ns', {}).get('db', 'Unknown') collection = event_data.get('ns', {}).get('coll', 'Unknown') full_document = event_data.get('fullDocument', {}) print(f"Operation type: {operation_type}") print(f"db: {db}") print(f"collection: {collection}") print("Full document:", json.dumps(full_document, indent=2))
Ruby
SDK for Ruby
Note

There's more on GitHub. Find the complete example and learn how to set up and run in the Serverless examples repository.

Consuming a Amazon DocumentDB event with Lambda using Ruby.

require 'json' def lambda_handler(event:, context:) event['events'].each do |record| log_document_db_event(record) end 'OK' end def log_document_db_event(record) event_data = record['event'] || {} operation_type = event_data['operationType'] || 'Unknown' db = event_data.dig('ns', 'db') || 'Unknown' collection = event_data.dig('ns', 'coll') || 'Unknown' full_document = event_data['fullDocument'] || {} puts "Operation type: #{operation_type}" puts "db: #{db}" puts "collection: #{collection}" puts "Full document: #{JSON.pretty_generate(full_document)}" end
To create the Lambda function
  1. Copy the sample code into a file named index.js.

  2. Create a deployment package with the following command.

    zip function.zip index.js
  3. Use the following CLI command to create the function. Replace us-east-1 with the region, and 123456789012 with your account ID.

    aws lambda create-function --function-name ProcessDocumentDBRecords \ --zip-file fileb://function.zip --handler index.handler --runtime nodejs20.x \ --region us-east-1 \ --role arn:aws:iam::123456789012:role/AWSDocumentDBLambdaExecutionRole

Create the Lambda event source mapping

Step 11 create the Lambda event source mapping

Create the event source mapping that associates your DocumentDB change stream with your Lambda function. After you create this event source mapping, Amazon Lambda immediately starts polling the stream.

To create the event source mapping
  1. Open the Functions page in the Lambda console.

  2. Choose the ProcessDocumentDBRecords function you created earlier.

  3. Choose the Configuration tab, then choose Triggers in the left menu.

  4. Choose Add trigger.

  5. Under Trigger configuration, for the source, select DocumentDB.

  6. Create the event source mapping with the following configuration:

    • DocumentDB cluster – Choose the cluster you created earlier.

    • Database namedocdbdemo

    • Collection name – products

    • Batch size – 1

    • Starting position – Latest

    • Authentication – BASIC_AUTH

    • Secrets Manager key – Choose the DocumentDBSecret you just created.

    • Batch window – 1

    • Full document configuration – UpdateLookup

  7. Choose Add. Creating your event source mapping can take a few minutes.

Test your function - manual invoke

Step 12 test your function with a manual invoke

To test that you created your function and event source mapping correctly, invoke your function using the invoke command. To do this, first copy the following event JSON into a file called input.txt:

{ "eventSourceArn": "arn:aws:rds:us-east-1:123456789012:cluster:canaryclusterb2a659a2-qo5tcmqkcl03", "events": [ { "event": { "_id": { "_data": "0163eeb6e7000000090100000009000041e1" }, "clusterTime": { "$timestamp": { "t": 1676588775, "i": 9 } }, "documentKey": { "_id": { "$oid": "63eeb6e7d418cd98afb1c1d7" } }, "fullDocument": { "_id": { "$oid": "63eeb6e7d418cd98afb1c1d7" }, "anyField": "sampleValue" }, "ns": { "db": "docdbdemo", "coll": "products" }, "operationType": "insert" } } ], "eventSource": "aws:docdb" }

Then, use the following command to invoke your function with this event:

aws lambda invoke --function-name ProcessDocumentDBRecords \ --cli-binary-format raw-in-base64-out \ --region us-east-1 \ --payload file://input.txt out.txt

You should see a response that looks like the following:

{ "StatusCode": 200, "ExecutedVersion": "$LATEST" }

You can verify that your function successfully processed the event by checking CloudWatch Logs.

To verify manual invocation via CloudWatch Logs
  1. Open the Functions page in the Lambda console.

  2. Choose the Monitor tab, then choose View CloudWatch logs. This takes you to the specific log group associated with your function in the CloudWatch console.

  3. Choose the most recent log stream. Within the log messages, you should see the event JSON.

Test your function - insert a record

Step 13 test your function by inserting a record.

Test your end-to-end setup by interacting directly with your DocumentDB database. In the next set of steps, you’ll insert a record, update it, then delete it.

To insert a record
  1. Reconnect to your DocumentDB cluster in your Cloud9 environment.

  2. Use this command to ensure that you’re currently using the docdbdemo database:

    use docdbdemo
  3. Insert a record into the products collection of the docdbdemo database:

    db.products.insert({"name":"Pencil", "price": 1.00})

Test your function - update a record

Step 14 test your function by updating a record.

Next, update the record you just inserted with the following command:

db.products.update( { "name": "Pencil" }, { $set: { "price": 0.50 }} )

Verify that your function successfully processed this event by checking CloudWatch Logs.

Test your function - delete a record

Step 15 test your function by deleting a record.

Finally, delete the record you just updated with the following command:

db.products.remove( { "name": "Pencil" } )

Verify that your function successfully processed this event by checking CloudWatch Logs.

Clean up your resources

You can now delete the resources that you created for this tutorial, unless you want to retain them. By deleting Amazon resources that you're no longer using, you prevent unnecessary charges to your Amazon Web Services account.

To delete the Lambda function
  1. Open the Functions page of the Lambda console.

  2. Select the function that you created.

  3. Choose Actions, Delete.

  4. Type delete in the text input field and choose Delete.

To delete the execution role
  1. Open the Roles page of the IAM console.

  2. Select the execution role that you created.

  3. Choose Delete.

  4. Enter the name of the role in the text input field and choose Delete.

To delete the VPC endpoints
  1. Open the VPC console. In the left menu, under Virtual private cloud, choose Endpoints.

  2. Select the endpoints you created.

  3. Choose Actions, Delete VPC endpoints.

  4. Enter delete in the text input field.

  5. Choose Delete.

To delete the Amazon DocumentDB cluster
  1. Open the DocumentDB console.

  2. Choose the DocumentDB cluster you created for this tutorial, and disable deletion protection.

  3. In the main Clusters page, choose your DocumentDB cluster again.

  4. Choose Actions, Delete.

  5. For Create final cluster snapshot, select No.

  6. Enter delete in the text input field.

  7. Choose Delete.

To delete the secret in Secrets Manager
  1. Open the Secrets Manager console.

  2. Choose the secret you created for this tutorial.

  3. Choose Actions, Delete secret.

  4. Choose Schedule deletion.

To delete the Amazon EC2 security group
  1. Open the EC2 console. Under Network and Security, choose Security groups.

  2. Select the security group you created for this tutorial.

  3. Choose Actions, Delete security groups.

  4. Choose Delete.

To delete the Cloud9 environment
  1. Open the Cloud9 console.

  2. Select the environment you created for this tutorial.

  3. Choose Delete.

  4. Enter delete in the text input field.

  5. Choose Delete.