REST API Connections - Amazon Glue
Creating a REST connection type in Amazon Glue
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

REST API Connections

Amazon Glue connectors cover a wide range of data sources both Amazon and external. However, there may be proprietary systems or emerging platforms that native Amazon Glue connectors do not support. The Amazon Glue REST connector provides a mechanism to configure a customized connector for any data source that has a REST-based API which then works the same way as natively supported Amazon Glue connectors. This capability minimizes the need to build custom libraries or ETL scripts to access these data sources.

To enable the REST API connector, Amazon Glue has introduced a new Amazon resource - Connection Type. The following operations can be used to manage connectivity to REST API-based data sources:

  • RegisterConnectionType - Registers a Connection Type in Amazon Glue based on the configuration provided. For request structure, See RegisterConnectionType.

  • DeleteConnectionType - Deletes a Connection Type configuration in Amazon Glue. See DeleteConnectionType.

  • DescribeConnectionType - Returns full details of the supported options for a given connection type in Amazon Glue. See DescribeConnectionType.

  • ListConnectionTypes - Returns list of connection types with high-level details of what is supported for each Amazon Glue connection type. See ListConnectionTypes.

Creating a REST connection type in Amazon Glue

To create a REST connection type, use the Amazon CLI, Amazon SDK or Amazon API to invoke the RegisterConnectionType API. Configuration details are provided based on the REST API being connected to.

Note

REST connections cannot be created using Amazon Glue console

Pre-Requisite

Before attempting to configure your Amazon Glue REST API connector, you need to familiarize yourself with the internal workings of the REST API. It helps if you have the REST API documentation for the data source and a tool for inspecting the request/ response structure.

The following properties of the REST API should be identified:

  • Authentication mechanism used for calls made to the REST endpoint. Amazon Glue REST API connectors support: basic authentication, OAuth2 - ClientCredentials, OAuth2 - Authorization Code, OAuth2 - JWT and custom authentication.

  • Entities (data objects) that you can transfer through the REST endpoint and related metadata (such as field names and data types).

  • The HTTP request and response structure. This structure includes the HTTP method, endpoint URL, headers, query parameters, and the format of the request body and response payload.

  • The pagination scheme. Amazon Glue REST API connectors support two pagination schemes: cursor-based and offset-based pagination.

RegisterConnectionType

Registers a ConnectionType in Amazon Glue based on the configuration provided. The ConnectionType resource is a new Amazon Glue resource that stores details about how requests and responses are interpreted by the data source. Today, only the REST protocol is supported. It encapsulates details about the REST API source such as:

  • Connection properties - Defines BaseUrl to connect to REST API and any additional request parameters needed to take input during Amazon Glue CreateConnection.

  • Authentication configuration - Defines how requests to the REST API are authenticated. This configuration is used when creating an Amazon Glue Connection for the Connection Type.

  • REST configuration - Defines HTTP request and response configuration to read data from REST API source. The following properties need to be configured at either the Global or Entity level:

    • Request method - Defines the REST HTTP Method.

    • Response configuration - Defines how the API response should be interpreted (including Records or Error locations in the body).

    • Pagination configuration - Defines the strategy that the REST API uses for paginating the resulting records and how the pagination properties are to be sent/ retrieved.

    • Validation endpoint configuration - Defines the endpoint to be called to validate an Amazon Glue connection.

    • Entity configurations - Defines the REST entities that the connector supports and their schema.

RegisterConnectionType Request Syntax

See RegisterConnectionType

Once a REST API connection type has been registered, it is ready to be used to create connections to the configured data source. Refer to Connecting to REST APIs for detailed instructions.