Creating the required VPC service endpoints in an Amazon VPC with private routing
An existing Amazon VPC network without Internet access needs additional VPC service endpoints (Amazon PrivateLink) to use Apache Airflow on Amazon Managed Workflows for Apache Airflow. This page describes the VPC endpoints required for the Amazon services used by Amazon MWAA, the VPC endpoints required for Apache Airflow, and how to create and attach the VPC endpoints to an existing Amazon VPC with private routing.
Contents
Pricing
Private network and private routing
The private network access mode limits access to the Apache Airflow UI to users within your Amazon VPC that have been granted access to the IAM policy for your environment.
When you create an environment with private web server access, you must package all of your dependencies in a Python wheel archive (.whl
), then
reference the .whl
in your requirements.txt
. For instructions on packaging and installing your dependencies
using wheel, see Managing dependencies using Python wheel.
The following image shows where to find the Private network option on the Amazon MWAA console.
-
Private routing. An Amazon VPC without Internet access limits network traffic within the VPC. This page assumes your Amazon VPC does not have Internet access and requires VPC endpoints for each Amazon service used by your environment, and VPC endpoints for Apache Airflow in the same Amazon Region and Amazon VPC as your Amazon MWAA environment.
(Required) VPC endpoints
The following section shows the required VPC endpoints needed for an Amazon VPC without Internet access. It lists the VPC endpoints for each Amazon service used by Amazon MWAA, including the VPC endpoints needed for Apache Airflow.
com.amazonaws.
YOUR_REGION
.s3 com.amazonaws.YOUR_REGION
.monitoring com.amazonaws.YOUR_REGION
.logs com.amazonaws.YOUR_REGION
.sqs com.amazonaws.YOUR_REGION
.kms
Note
When using Transit Gateway or any other routing that does not go directly to the Amazon API endpoints, we recommend you to add Amazon PrivateLink to your Amazon MWAA private subnets for the following services:
-
Amazon S3
-
Amazon SQS
-
CloudWatch Logs
-
CloudWatch metrics
-
Amazon KMS (if applicable)
This ensures that your Amazon MWAA environment can securely and efficiently communicate with these services without routing traffic through the public internet, thereby improving security and performance.
Attaching the required VPC endpoints
This section describes the steps to attach the required VPC endpoints for an Amazon VPC with private routing.
VPC endpoints required for Amazon services
The following section shows the steps to attach the VPC endpoints for the Amazon services used by an environment to an existing Amazon VPC.
To attach VPC endpoints to your private subnets
-
Open the Endpoints page
on the Amazon VPC console. -
Use the Amazon Region selector to select your region.
-
Create the endpoint for Amazon S3:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.s3
, then press Enter on your keyboard. -
We recommend choosing the service endpoint listed for the Gateway type.
For example,
com.amazonaws.us-west-2.s3 amazon Gateway
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that that private DNS is enabled by selecting Enable DNS name.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the endpoint for CloudWatch Logs:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.logs
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the endpoint for CloudWatch Monitoring:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.monitoring
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the endpoint for Amazon SQS:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.sqs
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the endpoint for Amazon KMS:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.kms
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
VPC endpoints required for Apache Airflow
The following section shows the steps to attach the VPC endpoints for Apache Airflow to an existing Amazon VPC.
To attach VPC endpoints to your private subnets
-
Open the Endpoints page
on the Amazon VPC console. -
Use the Amazon Region selector to select your region.
-
Create the endpoint for the Apache Airflow API:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.airflow.api
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the first endpoint for the Apache Airflow environment:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.airflow.env
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
-
Create the second endpoint for Apache Airflow operations:
-
Choose Create Endpoint.
-
In the Filter by attributes or search by keyword text field, type:
.airflow.ops
, then press Enter on your keyboard. -
Select the service endpoint.
-
Choose your environment's Amazon VPC in VPC.
-
Ensure that your two private subnets in different Availability Zones are selected, and that Enable DNS name is enabled.
-
Choose your environment's Amazon VPC security group(s).
-
Choose Full Access in Policy.
-
Choose Create endpoint.
-
(Optional) Enable private IP addresses for your Amazon S3 VPC interface endpoint
Amazon S3 Interface endpoints don't support private DNS. The S3 endpoint requests still resolves to a public IP address. To resolve the S3 address to a private IP address, you need to add a private hosted zone in Route 53 for the S3 regional endpoint.
Using Route 53
This section describes the steps to enable private IP addresses for an S3 Interface endpoint using Route 53.
-
Create a Private Hosted Zone for your Amazon S3 VPC interface endpoint (such as, s3.eu-west-1.amazonaws.com) and associate it with your Amazon VPC.
-
Create an ALIAS A record for your Amazon S3 VPC interface endpoint (such as, s3.eu-west-1.amazonaws.com) that resolves to your VPC Interface Endpoint DNS name.
-
Create an ALIAS A wildcard record for your Amazon S3 interface endpoint (such as, *.s3.eu-west-1.amazonaws.com) that resolves to the VPC Interface Endpoint DNS name.
VPCs with custom DNS
If your Amazon VPC uses custom DNS routing, you need to make the changes in your DNS resolver (not Route 53, typically an EC2 instance running a DNS server) by creating a CNAME record. For example:
Name: s3.us-west-2.amazonaws.com Type: CNAME Value: *.vpce-0f67d23e37648915c-e2q2e2j3.s3.us-west-2.vpce.amazonaws.com