Tutorial: Automate managing your own environment endpoints on Amazon MWAA - Amazon Managed Workflows for Apache Airflow
PrerequisitesCreate the Amazon VPCCreate the Lambda functionCreate the EventBridge ruleCreate the environment
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Tutorial: Automate managing your own environment endpoints on Amazon MWAA

If you use Amazon Organizations to manage multiple Amazon accounts that share resources, Amazon MWAA lets you create, and manage, your own Amazon VPC endpoints. This means you can use stricter security policies that allow access only the resources required by your environment.

When you create an environment in a shared Amazon VPC, the account that owns the main Amazon VPC (owner) shares the two private subnets required by Amazon MWAA with other accounts (participants) that belong to the same organization. Participant accounts that share those subnets can then view, create, modify, and delete environments in the shared VPC.

When you create an environment in a shared, or otherwise policy-restricted, Amazon VPC, Amazon MWAA will first create the service VPC resources, then enter a PENDING state for up to 72 hours.

When the environment status changes from CREATING to PENDING, Amazon MWAA sends an Amazon EventBridge notification of the change in state. This lets the owner account create the required endpoints on behalf of participants based on endpoint service information from the Amazon MWAA console or API, or programmatically In the following, we create new Amazon VPC endpoints using an Lambda function and an EventBridge rule that listens to Amazon MWAA state change notifications.

Here, we create the new endpoints in the same Amazon VPC as the environment. To set up a shared Amazon VPC, create the EventBridge rule and Lambda function would in the owner account, and the Amazon MWAA environment in the participant account.

Prerequisites

To complete the steps in this tutorial, you will need the following:

  • ...

Create the Amazon VPC

Use the following Amazon CloudFormation template and Amazon CLI command to create a new Amazon VPC. The template sets up the Amazon VPC resources and modifies the endpoint policy to restrict access to a specific queue.

  1. Download the Amazon CloudFormation template, then unzip the .yml file.

  2. In a new command prompt window, navigate to the folder where you saved the template, then use create-stack to create the stack. The --template-body flag specifies the path to the template.

    $ aws cloudformation create-stack --stack-name stack-name --template-body file://cfn-vpc-private-network.yml

In the next section, you'll create the Lambda function.

Create the Lambda function

Use the following Python code and IAM JSON policy to create a new Lambda function and execution role. This function creates Amazon VPC endpoints for a private Apache Airflow web server and an Amazon SQS queue. Amazon MWAA uses Amazon SQS to queue tasks with Celery among multiple workers when scaling your environment.

  1. Download the Python function code.

  2. Download the IAM permission policy, then unzip the file.

  3. Open a command prompt, then navigate to the folder where you saved the JSON permission policy. Use the IAM create-role command to create the new role. 

    $ aws iam create-role --role-name function-role \
--assume-role-policy-document file://lambda-mwaa-vpce-policy.json

    Note the role ARN from the Amazon CLI response. In the next step, we specify this new role as the function's execution role using its ARN.

  4. Navigate to the folder where you saved the function code, then use thecreate-function command to create a new function.

    $ aws lambda create-function --function-name mwaa-vpce-lambda \
--zip-file file://mwaa-lambda-shared-vpc.zip --runtime python3.8 --role arn:aws:iam::123456789012:role/function-role --handler lambda_handler

    Note the function ARN from the Amazon CLI response. In the next step we specify the ARN to configure the function as a target for a new EventBridge rule.

In the next section, you will create the EventBridge rule that invokes this function when the environment enters a PENDING state.

Create the EventBridge rule

Do the following to create a new rule that listens for Amazon MWAA notifications and targets your new Lambda function.

  1. Use the EventBridge put-rule command to create a new EventBridge rule. 

    $ aws events put-rule --name "mwaa-lambda-rule" \
--event-pattern "{\"source\":[\"aws.airflow\"],\"detail-type\":[\"MWAA Environment Status Change\"]}"

    The event pattern listens for notifications that Amazon MWAA sends whenever an environment status changes. 

    {
  "source": ["aws.airflow"],
  "detail-type": ["MWAA Environment Status Change"]
}

  2. Use the put-targets command to add the Lambda function as a target for the new rule. 

    $ aws events put-targets --rule "mwaa-lambda-rule" \
--targets "Id"="1","Arn"="arn:aws-cn::lambda:region:123456789012:function:mwaa-vpce-lambda"

You're ready to create a new Amazon MWAA environment with customer-managed Amazon VPC endpoints.

Create the Amazon MWAA environment

Use the Amazon MWAA console to create a new environment with customer-managed Amazon VPC endpoints.

  1. Open the Amazon MWAA console, and choose Create an environment.

  2. For Name enter a unique name.

  3. For Airflow version choose the latest version.

  4. Choose an Amazon S3 bucket and a DAGs folder, such as dags/ to use with the environment, then choose Next.

  5. On the Configure advanced settings page, do the following:

    1. For Virtual Private Cloud, choose the Amazon VPC you created in the previous step.

    2. For Web server access, choose Public network (Internet accessible).

    3. For Security groups, choose the security group you created with Amazon CloudFormation. Because the security groups for the Amazon PrivateLink endpoints from the earlier step are self-referencing, you must choose the same security group for your environment.

    4. For Endpoint management, choose Customer managed endpoints.

  6. Keep the remaining default settings, then choose Next.

  7. Review your selections, then choose Create environment.

Tip

For more information about setting up a new environment, see Getting started with Amazon MWAA.

When the environment is PENDING, Amazon MWAA sends a notification that matches the event pattern you set for your rule. The rule invokes your Lambda function. The function parses the notification event and gets the required endpoint information for the web server and the Amazon SQS queue. It then creates the endpoints in your Amazon VPC.

When the endpoints are available, Amazon MWAA resumes creating your environment. When ready, the environment status changes to AVAILABLE and you can access the Apache Airflow web server using the Amazon MWAA console.