# Managing connections to Apache Airflow Managing connections This chapter describes how to configure an Apache Airflow connection for an Amazon Managed Workflows for Apache Airflow environment. **Topics** + [ ## Overview of Apache Airflow variables and connections ](#manage-connections-t-overview) + [ # Apache Airflow provider packages installed on Amazon MWAA environments ](connections-packages.md) + [ # Overview of connection types ](manage-connection-types.md) + [ # Configuring an Apache Airflow connection using a Amazon Secrets Manager secret ](connections-secrets-manager.md) ## Overview of Apache Airflow variables and connections Overview In some cases, you might want to specify additional connections or variables for an environment, such as an Amazon profile, or to add your execution role in a connection object in the Apache Airflow metastore, then refer to the connection from within a DAG. + **Self-managed Apache Airflow**. On a self-managed Apache Airflow installation, you set [Apache Airflow configuration options in `airflow.cfg`](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html). ``` [secrets] backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend backend_kwargs = {"connections_prefix" : "airflow/connections", "variables_prefix" : "airflow/variables"} ``` + **Apache Airflow on Amazon MWAA**. On Amazon MWAA, you need to add these configuration settings as [Apache Airflow configuration options](configuring-env-variables.md) on the Amazon MWAA console. Apache Airflow configuration options are written as environment variables to your environment and override all other existing configurations for the same setting. # Apache Airflow provider packages installed on Amazon MWAA environments Apache Airflow packages This page lists the Apache Airflow provider packages installed by Amazon MWAA for all supported Apache Airflow environments. For more information about these packages, refer to the [Apache Airflow reference for package extras](https://airflow.apache.org/docs/apache-airflow/stable/extra-packages-ref.html). **Note** To ensure that compatibility with CloudWatch logging is not overridden by other Python library installations, Amazon MWAA installs [Watchtower version 2.0.1](https://pypi.org/project/watchtower/2.0.1/) after performing `pip3 install -r requirements.txt`. **Topics** + [ ## Constraints file ](#connections-packages-constraints) + [ ## Version-specific provider packages ](#connections-packages-table) ## Constraints file Beginning with Apache Airflow v2.7.2, your requirements file must include a `--constraint` statement. If you don't provide a constraint, Amazon MWAA will specify one for you to ensure the packages listed in your requirements are compatible with the version of Apache Airflow you're using. Apache Airflow constraints files specify the provider versions available at the time of an Apache Airflow release. In many cases, however, newer providers are compatible with that version of Apache Airflow. Because you must use constraints, to specify a newer version of a provider package, you can modify the constraints file for a specific provider version: 1. Download the version-specific constraints file from GitHub, for example [https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt](https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.11.txt) (replace '2.7.2' with the version you want to use). 1. Save the modified constraints file to the Amazon S3 dags folder of your Amazon MWAA environment, for example, as `constraints-3.11-updated.txt`. 1. Specify your requirements as listed in the following. ``` --constraint "/usr/local/airflow/dags/constraints-3.11-updated.txt" apache-airflow-providers-amazon==version-number ``` **Note** If you are using a private webserver, we recommend that you [package the required libraries as WHL files](best-practices-dependencies.md#best-practices-dependencies-python-wheels) by using [aws-mwaa-docker-images](https://github.com/aws/amazon-mwaa-docker-images). ## Version-specific provider packages Installing provider packages you can use to access a connection type in the Apache Airflow UI. It also means you don't need to specify these packages as a Python dependency in your `requirements.txt` file. This page lists the Apache Airflow provider packages installed by Amazon MWAA for all supported Apache Airflow environments. **Note** For Apache Airflow v2 and later, Amazon MWAA installs [Watchtower version 2.0.1](https://pypi.org/project/watchtower/2.0.1/) after performing `pip3 install -r requirements.txt`, to ensure compatibility with CloudWatch logging is not overridden by other Python library installations. You can specify the latest supported version of `apache-airflow-providers-amazon` to upgrade this provider. **Supported Apache Airflow versions:** ------ #### [ v3.0.6 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==9.9.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.9.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==6.2.1](https://airflow.apache.org/docs/apache-airflow-providers-postgres/6.2.1/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.13.1](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.13.1/index.html) | | Fab Connection | [apache-airflow-providers-fab==2.3.0](https://airflow.apache.org/docs/apache-airflow-providers-fab/2.3.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.12.1](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.12.1/index.html) | | HTTP Connection | [apache-airflow-providers-http==5.3.2](https://airflow.apache.org/docs/apache-airflow-providers-http/5.3.2/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.9.1](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.9.1/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.27.3](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.3/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==4.1.1](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/4.1.1/index.html) | ------ #### [ v2.11.0 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==9.8.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.8.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==6.2.0](https://airflow.apache.org/docs/apache-airflow-providers-postgres/6.2.0/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.13.0](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.13.0/index.html) | | Fab Connection | [apache-airflow-providers-fab==1.5.3](https://airflow.apache.org/docs/apache-airflow-providers-fab/1.5.3/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.11.0](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.11.0/index.html) | | HTTP Connection | [apache-airflow-providers-http==5.3.0](https://airflow.apache.org/docs/apache-airflow-providers-http/5.3.0/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.9.0](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.9.0/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.27.1](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.27.1/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==4.1.0](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/4.1.0/index.html) | | SMTP Connection | [apache-airflow-providers-smtp==2.1.0](https://airflow.apache.org/docs/apache-airflow-providers-smtp/2.1.0/index.html) | ------ #### [ v2.10.3 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==9.0.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/9.0.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.13.1](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.13.1/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.11.1](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.11.1/index.html) | | Fab Connection | [apache-airflow-providers-fab==1.5.0](https://airflow.apache.org/docs/apache-airflow-providers-fab/1.5.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.8.3](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.8.3/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.13.2](https://airflow.apache.org/docs/apache-airflow-providers-http/4.13.2/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.7.0](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.7.0/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.19.0](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.19.0/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.9.0](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.9.0/index.html) | | SMTP Connection | [apache-airflow-providers-smtp==1.8.0](https://airflow.apache.org/docs/apache-airflow-providers-smtp/1.8.0/index.html) | ------ #### [ v2.10.1 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==8.28.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.28.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.12.0](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.12.0/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.11.0](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.11.0/index.html) | | Fab Connection | [apache-airflow-providers-fab==1.3.0](https://airflow.apache.org/docs/apache-airflow-providers-fab/1.3.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.8.1](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.8.1/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.13.0](https://airflow.apache.org/docs/apache-airflow-providers-http/4.13.0/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.7.0](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.7.0/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.16.0](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.16.0/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.9.0](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.9.0/index.html) | | SMTP Connection | [apache-airflow-providers-smtp==1.8.0](https://airflow.apache.org/docs/apache-airflow-providers-smtp/1.8.0/index.html) | ------ #### [ v2.9.2 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==8.24.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.24.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.11.1](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.11.1/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.9.1](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.9.1/index.html) | | Fab Connection | [apache-airflow-providers-fab==1.1.1](https://airflow.apache.org/docs/apache-airflow-providers-fab/1.1.1/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.7.2](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.7.2/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.11.1](https://airflow.apache.org/docs/apache-airflow-providers-http/4.11.1/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.6.1](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.6.1/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.14.0](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.14.0/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.8.1](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.8.1/index.html) | | SMTP Connection | [apache-airflow-providers-smtp==1.7.1](https://airflow.apache.org/docs/apache-airflow-providers-smtp/1.7.1/index.html) | ------ #### [ v2.8.1 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==8.16.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.16.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.10.0](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.10.0/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.7.0](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.7.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.5.1](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.5.1/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.8.0](https://airflow.apache.org/docs/apache-airflow-providers-http/4.8.0/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.5.0](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.5.0/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.10.0](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.10.0/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.7.0](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.7.0/index.html) | ------ #### [ v2.7.2 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==8.7.1](https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.7.1/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.6.1](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.6.1/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.5.2](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.5.2/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.3.4](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.3.4/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.5.2](https://airflow.apache.org/docs/apache-airflow-providers-http/4.5.2/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.3.2](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.3.2/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.7.2](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.7.2/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.4.3](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.4.3/index.html) | ------ #### [ v2.6.3 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon[aiobotocore]==8.2.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/8.2.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.5.1](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.5.1/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.4.2](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.4.2/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.2.1](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.2.1/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.4.2](https://airflow.apache.org/docs/apache-airflow-providers-http/4.4.2/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.2.2](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.2.2/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.5.2](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.5.2/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.4.2](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.4.2/index.html) | ------ #### [ v2.5.1 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon==7.1.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/7.1.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.4.0](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.4.0/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.3.0](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.3.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.1.0](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.1.0/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.1.1](https://airflow.apache.org/docs/apache-airflow-providers-http/4.4.1/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.1.1](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.1.1/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.3.3](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.3.3/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.3.1](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.3.1/index.html) | ------ #### [ v2.4.3 ] | Connection type | Package | | --- | --- | | Amazon Connection | [apache-airflow-providers-amazon==6.0.0](https://airflow.apache.org/docs/apache-airflow-providers-amazon/6.0.0/index.html) | | Postgres Connection | [apache-airflow-providers-postgres==5.2.2](https://airflow.apache.org/docs/apache-airflow-providers-postgres/5.2.2/index.html) | | FTP Connection | [apache-airflow-providers-ftp==3.1.0](https://airflow.apache.org/docs/apache-airflow-providers-ftp/3.1.0/index.html) | | Celery Connection | [apache-airflow-providers-celery==3.0.0](https://airflow.apache.org/docs/apache-airflow-providers-celery/3.0.0/index.html) | | HTTP Connection | [apache-airflow-providers-http==4.0.0](https://airflow.apache.org/docs/apache-airflow-providers-http/4.0.0/index.html) | | IMAP Connection | [apache-airflow-providers-imap==3.0.0](https://airflow.apache.org/docs/apache-airflow-providers-imap/3.0.0/index.html) | | Common SQL | [apache-airflow-providers-common-sql==1.2.0](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/1.2.0/index.html) | | SQLite Connection | [apache-airflow-providers-sqlite==3.2.1](https://airflow.apache.org/docs/apache-airflow-providers-sqlite/3.2.1/index.html) | ------ # Overview of connection types Connection types Apache Airflow stores connections as a connection URI string. It provides a connections template in the Apache Airflow UI to generate the connection URI string, regardless of the connection type. If a connection template is not available in the Apache Airflow UI, an alternate connection template can be used to generate this connection URI string, such as using the HTTP connection template. The primary difference is the URI prefix, such as `my-conn-type://`, which Apache Airflow providers typically ignore for a connection. This page describes how to use connection templates in the Apache Airflow UI interchangeably for different connection types. **Warning** Don't overwrite the [https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) connection in Amazon MWAA. Amazon MWAA uses this connection to perform a variety of critical tasks, such as collecting task logs. Overwriting this connection might result in data loss and disruptions to your environment availability. **Topics** + [ ## Example connection URI string ](#manage-connection-types-string-example) + [ ## Example connection template ](#manage-connection-types-template-example) + [ ## Example using an HTTP connection template for a Jdbc connection ](#manage-connection-types-example) ## Example connection URI string The following example presents a connection URI string for the MySQL connection type. ``` 'mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA®ion_name=us-east-1' ``` ## Example connection template The following examples depict the HTTP connection template in the Apache Airflow UI. ------ #### [ Apache Airflow v3 ] ![\[This image depicts the HTTP connection type in the Apache Airflow UI.\]](http://docs.amazonaws.cn/en_us/mwaa/latest/userguide/images/airflow-connection-v3.0.6.png) ------ #### [ Apache Airflow v2 ] ![\[This image depicts the HTTP connection type in the Apache Airflow UI.\]](http://docs.amazonaws.cn/en_us/mwaa/latest/userguide/images/airflow-connection-v2.0.2.png) ------ ## Example using an HTTP connection template for a Jdbc connection Use the following example to apply the **HTTP** connection template for a *Jdbc* connection type in the Apache Airflow UI. ------ #### [ Apache Airflow v3 ] The following example displays the connection URI string generated by Apache Airflow for the example in this section. ``` http://myconnectionurl/some/path&login=mylogin&extra__jdbc__dry__path=usr/local/airflow/dags/classpath/redshif-jdbc42-2.0.0.1.jar&extra__jdbc__dry__clsname=redshift-jdbc42-2.0.0.1 ``` Use the following example to apply the HTTP connection template for a *Jdbc* connection for Apache Airflow v3 in the Apache Airflow UI. ![\[This image depicts the HTTP connection type for a Jdbc connection in the Apache Airflow UI.\]](http://docs.amazonaws.cn/en_us/mwaa/latest/userguide/images/airflow-connection-jdbc-v3.0.6.png) ------ #### [ Apache Airflow v2 ] The following example displays the connection URI string generated by Apache Airflow for the example in this section. ``` http://myconnectionurl/some/path&login=mylogin&extra__jdbc__dry__path=usr/local/airflow/dags/classpath/redshif-jdbc42-2.0.0.1.jar&extra__jdbc__dry__clsname=redshift-jdbc42-2.0.0.1 ``` Use the following example to apply the HTTP connection template for a *Jdbc* connection for Apache Airflow v2 in the Apache Airflow UI. ![\[This image depicts the HTTP connection type for a Jdbc connection in the Apache Airflow UI.\]](http://docs.amazonaws.cn/en_us/mwaa/latest/userguide/images/airflow-connection-jdbc-v2.0.2.png) ------ # Configuring an Apache Airflow connection using a Amazon Secrets Manager secret Configuring Secrets Manager Amazon Secrets Manager is a supported alternative Apache Airflow backend on an Amazon Managed Workflows for Apache Airflow environment. This topic explains how to use Amazon Secrets Manager to securely store secrets for Apache Airflow variables and an Apache Airflow connection on Amazon Managed Workflows for Apache Airflow. **Note** You are charged for the secrets you create. For more information about Secrets Manager pricing, refer to [Amazon Pricing](https://www.amazonaws.cn/secrets-manager/pricing/). [Amazon Systems Manager Parameter Store](https://docs.amazonaws.cn/systems-manager/latest/userguide/systems-manager-parameter-store) is also supported as a secrets backend in Amazon MWAA. For more information, refer to [Amazon Provider Package documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-ssm-parameter-store.html). **Contents** + [ ## Step one: Provide Amazon MWAA with permission to access Secrets Manager secret keys ](#connections-sm-policy) + [ ## Step two: Create the Secrets Manager backend as an Apache Airflow configuration option ](#connections-sm-aa-configuration) + [ ## Step three: Generate an Apache Airflow Amazon connection URI string ](#connections-sm-aa-uri) + [ ## Step four: Add the variables in Secrets Manager ](#connections-sm-createsecret-variables) + [ ## Step five: Add the connection in Secrets Manager ](#connections-sm-createsecret-connection) + [ ## Sample code ](#connections-sm-samples) + [ ## Resources ](#connections-sm-blogs) + [ ## What's next? ](#connections-sm-next-up) ## Step one: Provide Amazon MWAA with permission to access Secrets Manager secret keys The [execution role](mwaa-create-role.md) for your Amazon MWAA environment needs read access to the secret key in Amazon Secrets Manager. The following IAM policy allows read-write access using the Amazon-managed [SecretsManagerReadWrite](https://console.amazonaws.cn/iam/home?#/policies/arn:aws:iam::aws:policy/SecretsManagerReadWrite$jsonEditor) policy. **To attach the policy to your execution role** 1. Open the [Environments](https://console.amazonaws.cn/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose your execution role on the **Permissions** pane. 1. Choose **Attach policies**. 1. Type `SecretsManagerReadWrite` in the **Filter policies** text field. 1. Choose **Attach policy**. If you do not want to use an Amazon-managed permission policy, you can directly update your environment's execution role to allow any level of access to your Secrets Manager resources. For example, the following policy statement grants read access to all secrets you create in a specific Amazon Web Services Region in Secrets Manager. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds" ], "Resource": "arn:aws-cn:secretsmanager:us-east-1:111122223333:secret:*" }, { "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" } ] } ``` ------ ## Step two: Create the Secrets Manager backend as an Apache Airflow configuration option The following section describes how to create an Apache Airflow configuration option on the Amazon MWAA console for the Amazon Secrets Manager backend. If you're using a configuration setting of the same name in `airflow.cfg`, the configuration you create in the following steps takes precedence and override the configuration settings. 1. Open the [Environments](https://console.amazonaws.cn/mwaa/home#/environments) page on the Amazon MWAA console. 1. Choose an environment. 1. Choose **Edit**. 1. Choose **Next**. 1. Choose **Add custom configuration** in the **Airflow configuration options** pane. Add the following key-value pairs: 1. **`secrets.backend`**: **`airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend`** 1. **`secrets.backend_kwargs`**: **`{"connections_prefix" : "airflow/connections", "variables_prefix" : "airflow/variables"}`** This configures Apache Airflow to search for connection strings and variables at `airflow/connections/*` and `airflow/variables/*` paths. You can use a [lookup pattern](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html#optional-lookup) to reduces the number of API calls Amazon MWAA makes to Secrets Manager on your behalf. If you do not specify a lookup pattern, Apache Airflow searches for all connections and variables in the configured backend. By specifying a pattern, you narrow the possible paths that Apache Airflow searches. This lowers your costs when using Secrets Manager with Amazon MWAA. To specify a lookup pattern, specify the `connections_lookup_pattern` and `variables_lookup_pattern` parameters. These parameters accept a RegEx string as input. For example, to search for secrets that start with `test`, enter the following for `secrets.backend_kwargs`: ``` { "connections_prefix": "airflow/connections", "connections_lookup_pattern": "^test", "variables_prefix" : "airflow/variables", "variables_lookup_pattern": "^test" } ``` **Note** To use `connections_lookup_pattern` and `variables_lookup_pattern`, you must install `apache-airflow-providers-amazon` version 7.3.0 or higher. For more information about updating provder pacakges for to newer versions, refer to [Constraints file](connections-packages.md#connections-packages-constraints). 1. Choose **Save**. ## Step three: Generate an Apache Airflow Amazon connection URI string To create a connection string, use the "tab" key on your keyboard to indent the key-value pairs in the [Connection](https://airflow.apache.org/docs/stable/howto/connection/index.html) object. We also recommend creating a variable for the `extra` object in your shell session. The following section walks you through the steps to [generate an Apache Airflow connection URI](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri) string for an Amazon MWAA environment using Apache Airflow or a Python script. ------ #### [ Apache Airflow CLI ] The following shell session uses your local Airflow CLI to generate a connection string. If you don't have the CLI installed, we recommend using the Python script. 1. Open a Python shell session: ``` python3 ``` 1. Enter the following command: ``` >>> import json ``` 1. Enter the following command: ``` >>> from airflow.models.connection import Connection ``` 1. Create a variable in your shell session for the `extra` object. Substitute the sample values in *YOUR\$1EXECUTION\$1ROLE\$1ARN* with the execution role ARN, and the region in *us-east-1* (such as `us-east-1`). ``` >>> extra=json.dumps({'role_arn': 'YOUR_EXECUTION_ROLE_ARN', 'region_name': 'us-east-1'}) ``` 1. Create the connection object. Substitute the sample value in `myconn` with the name of the Apache Airflow connection. ``` >>> myconn = Connection( ``` 1. Use the "tab" key on your keyboard to indent each of the following key-value pairs in your connection object. Substitute the sample values in *red*. 1. Specify the Amazon connection type: ``` ... conn_id='aws', ``` 1. Specify the Apache Airflow database option: ``` ... conn_type='mysql', ``` 1. Specify the Apache Airflow UI URL on Amazon MWAA: ``` ... host='288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com/home', ``` 1. Specify the Amazon access key ID (username) to log in to Amazon MWAA: ``` ... login='YOUR_AWS_ACCESS_KEY_ID', ``` 1. Specify the Amazon secret access key (password) to log in to Amazon MWAA: ``` ... password='YOUR_AWS_SECRET_ACCESS_KEY', ``` 1. Specify the `extra` shell session variable: ``` ... extra=extra ``` 1. Close the connection object. ``` ... ) ``` 1. Print the connection URI string: ``` >>> myconn.get_uri() ``` Refer to the connection URI string in the response: ``` 'mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA®ion_name=us-east-1' ``` ------ #### [ Python script ] The following Python script does not require the Apache Airflow CLI. 1. Copy the contents of the following code sample and save locally as `mwaa_connection.py`. ``` import urllib.parse conn_type = 'YOUR_DB_OPTION' host = 'YOUR_MWAA_AIRFLOW_UI_URL' port = 'YOUR_PORT' login = 'YOUR_AWS_ACCESS_KEY_ID' password = 'YOUR_AWS_SECRET_ACCESS_KEY' role_arn = urllib.parse.quote_plus('YOUR_EXECUTION_ROLE_ARN') region_name = 'us-east-1' conn_string = '{0}://{1}:{2}@{3}:{4}?role_arn={5}®ion_name={6}'.format(conn_type, login, password, host, port, role_arn, region_name) print(conn_string) ``` 1. Substitute the placeholders in *red*. 1. Run the following script to generate a connection string. ``` python3 mwaa_connection.py ``` ------ ## Step four: Add the variables in Secrets Manager The following section describes how to create the secret for a variable in Secrets Manager. **To create the secret** 1. Open the [Amazon Secrets Manager console](https://console.amazonaws.cn/secretsmanager/home#/environments). 1. Choose **Store a new secret**. 1. Choose **Other type of secret**. 1. On the **Specify the key/value pairs to be stored in this secret** pane, choose **Plaintext**. 1. Add the variable value as **Plaintext** in the following format. ``` "YOUR_VARIABLE_VALUE" ``` For example, to specify an integer: ``` 14 ``` For example, to specify a string: ``` "mystring" ``` 1. For **Encryption key**, choose an Amazon KMS key option from the dropdown list. 1. Enter a name in the text field for **Secret name** in the following format. ``` airflow/variables/YOUR_VARIABLE_NAME ``` For example: ``` airflow/variables/test-variable ``` 1. Choose **Next**. 1. On the **Configure secret** page, on the **Secret name and description** pane, do the following. 1. For **Secret name**, provide a name for your secret. 1. (Optional) For **Description**, provide a description for your secret. Choose **Next**. 1. On the **Configure rotation - optional** leave the default options and choose **Next**. 1. Repeat these steps in Secrets Manager for any additional variables you want to add. 1. On the **Review** page, review your secret, then choose **Store**. ## Step five: Add the connection in Secrets Manager The following section describes how to create the secret for your connection string URI in Secrets Manager. **To create the secret** 1. Open the [Amazon Secrets Manager console](https://console.amazonaws.cn/secretsmanager/home#/environments). 1. Choose **Store a new secret**. 1. Choose **Other type of secret**. 1. On the **Specify the key/value pairs to be stored in this secret** pane, choose **Plaintext**. 1. Add the connection URI string as **Plaintext** in the following format. ``` YOUR_CONNECTION_URI_STRING ``` For example: ``` mysql://288888a0-50a0-888-9a88-1a111aaa0000.a1.us-east-1.airflow.amazonaws.com%2Fhome?role_arn=arn%3Aaws%3Aiam%3A%3A001122332255%3Arole%2Fservice-role%2FAmazonMWAA-MyAirflowEnvironment-iAaaaA®ion_name=us-east-1 ``` **Warning** Apache Airflow parses each of the values in the connection string. You must **not** use single nor double quotes, or it parses the connection as a single string. 1. For **Encryption key**, choose an Amazon KMS key option from the dropdown list. 1. Enter a name in the text field for **Secret name** in the following format. ``` airflow/connections/YOUR_CONNECTION_NAME ``` For example: ``` airflow/connections/myconn ``` 1. Choose **Next**. 1. On the **Configure secret** page, on the **Secret name and description** pane, do the following. 1. For **Secret name**, provide a name for your secret. 1. (Optional) For **Description**, provide a description for your secret. Choose **Next**. 1. On the **Configure rotation - optional** leave the default options and choose **Next**. 1. Repeat these steps in Secrets Manager for any additional variables you want to add. 1. On the **Review** page, review your secret, then choose **Store**. ## Sample code + Learn how to use the secret key for the Apache Airflow connection (`myconn`) on this page using the sample code at [Using a secret key in Amazon Secrets Manager for an Apache Airflow connection](samples-secrets-manager.md). + Learn how to use the secret key for the Apache Airflow variable (`test-variable`) on this page using the sample code at [Using a secret key in Amazon Secrets Manager for an Apache Airflow variable](samples-secrets-manager-var.md). ## Resources + For more information about configuring Secrets Manager secrets using the console and the Amazon CLI, refer to [Create a secret](https://docs.amazonaws.cn/secretsmanager/latest/userguide/create_secret.html) in the *Amazon Secrets Manager User Guide*. + Use a Python script to migrate a large volume of Apache Airflow variables and connections to Secrets Manager in [Move your Apache Airflow connections and variables to Amazon Secrets Manager](https://aws.amazon.com/blogs/opensource/move-apache-airflow-connections-variables-aws-secrets-manager/). ## What's next? + Learn how to generate a token to access the Apache Airflow UI in [Accessing Apache Airflow](access-airflow-ui.md).