Connecting to data - Amazon Glue
Overview of using connectors and connections
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Connecting to data

An Amazon Glue connection is a Data Catalog object that stores login credentials, URI strings, virtual private cloud (VPC) information, and more for a particular data store. Amazon Glue crawlers, jobs, and development endpoints use connections in order to access certain types of data stores. You can use connections for both sources and targets, and reuse the same connection across multiple crawler or extract, transform, and load (ETL) jobs.

The latest version of the Amazon Glue connections schema provides a unified way to manage data connections across Amazon services and applications, such as Amazon Glue, Amazon Athena, and Amazon SageMaker AI Unified Studio.

Overview of using connectors and connections

A connection contains the properties that are required to connect to a particular data store. When you create a connection, it is stored in the Amazon Glue Data Catalog. You choose a connector, and then create a connection based on that connector.

You can subscribe to connectors for non-natively supported data stores in Amazon Web Services Marketplace, and then use those connectors when you're creating connections. Developers can also create their own connectors, and you can use them when creating connections.

Note

Connections created using custom or Amazon Web Services Marketplace connectors in Amazon Glue Studio appear in the Amazon Glue console with type set to UNKNOWN.

The following steps describe the overall process of using connectors in Amazon Glue Studio:

  1. Subscribe to a connector in Amazon Web Services Marketplace, or develop your own connector and upload it to Amazon Glue Studio. For more information, see Adding connectors to Amazon Glue Studio.

  2. Review the connector usage information. You can find this information on the Usage tab on the connector product page. For example, if you click the Usage tab on this product page, Amazon Glue Connector for Google BigQuery, you can see in the Additional Resources section a link to a blog about using this connector.

  3. Create a connection. You choose which connector to use and provide additional information for the connection, such as login credentials, URI strings, and virtual private cloud (VPC) information. For more information, see Creating connections for connectors.

  4. Create an IAM role for your job. The job assumes the permissions of the IAM role that you specify when you create it. This IAM role must have the necessary permissions to authenticate with, extract data from, and write data to your data stores.

  5. Create an ETL job and configure the data source properties for your ETL job. Provide the connection options and authentication information as instructed by the custom connector provider. For more information, see Authoring jobs with custom connectors.

  6. Customize your ETL job by adding transforms or additional data stores, as described in Starting visual ETL jobs in Amazon Glue Studio.

  7. If using a connector for the data target, configure the data target properties for your ETL job. Provide the connection options and authentication information as instructed by the custom connector provider. For more information, see Authoring jobs with custom connectors.

  8. Customize the job run environment by configuring job properties, as described in Modify the job properties.

  9. Run the job.