Tutorial: Using the Amazon Glue Connector for Elasticsearch

Sign in to the Amazon Secrets Manager console.

On either the service introduction page or the Secrets list page, choose Store a new secret.

On the Store a new secret page, choose Other type of secret. This option means that you must supply the structure and details of your secret.

Add a Key and Value pair for the OpenSearch cluster user name. For example:

es.net.http.auth.user: username

Choose + Add row, and enter another key-value pair for the password. For example:

es.net.http.auth.pass: password

Choose Next.

Enter a secret name. For example: my-es-secret. You can optionally include a description.

Record the secret name, which is used later in this tutorial, and then choose Next.

Choose Next again, and then choose Store to create the secret.

If you have not already configured your Amazon account to use License Manager, do the following:

1. Open the Amazon License Manager console at https://console.amazonaws.cn/license-manager.

2. Choose Create customer managed license.

3. In the IAM permissions (one-time setup) window, choose I grant Amazon License Manager the required permissions, and then choose Grant permissions.

   If you do not see this window, then you have already configured the necessary permissions.

Open the Amazon Glue Studio console at https://console.amazonaws.cn/gluestudio/.

In the Amazon Glue Studio console, expand the menu icon ( ), and then choose Connectors in the navigation pane.

On the Connectors page, choose Go to Amazon Web Services Marketplace.

In Amazon Web Services Marketplace, in the Search Amazon Glue Studio products section, enter Amazon Glue Connector for Elasticsearch in the search field, and then press Enter.

Choose the name of the connector, Amazon Glue Connector for Elasticsearch.

On the product page for the connector, use the tabs to view information about the connector. When you're ready to continue, choose Continue to Subscribe.

Review the terms of use for the software. Click Accept Terms.

When the subscription process completes, you will see a notification: "Thank you for subscribing to this product! You can now configure your software." Above the banner will be the button Continue to Configuration. Choose Continue to Configuration.

Choose the Fulfillment option on the Configure this software page. You can either choose between Amazon Glue 1.0/2.0 or Amazon Glue 3.0. Then, choose Continue to Launch.

On the Launch this software page in the Amazon Web Services Marketplace console, choose Usage Instructions, and then choose the link in the window that appears.

Your browser is redirected to the Amazon Glue Studio console Create marketplace connection page.

Enter a name for the connection. For example: my-es-connection.

In the Connection access section, for Connection credential type, choose User name and password.

For the Amazon secret, enter the name of your secret. For example: my-es-secret.

In the Network options section, enter the VPC information to connect to OpenSearch cluster.

Choose Create connection and activate connector.

Configure the permissions described in Review IAM permissions needed for ETL jobs.

Configure the additional permissions needed when using connectors with Amazon Glue Studio, as described in Permissions required for using connectors.

In Amazon Glue Studio, choose Connectors.

In the Your connections list, select the connection you just created and choose Create job.

In the visual job editor, choose the Data source node. On the right, on the Data source properties - Connector tab, configure additional information for the connector.

1. Choose Add schema and enter the schema of the data set in the data source. Connections do not use tables stored in the Data Catalog, which means that Amazon Glue Studio doesn't know the schema of the data. You must manually provide this schema information. For instructions on how to use the schema editor, see Editing the schema in a custom transform node.

2. Expand Connection options.

3. Choose Add new option and enter the information needed for the connector that was not entered in the Amazon secret:

   • es.nodes: https://<OpenSearch domain endpoint>

   • es.port: 443

   • path: test

   • es.nodes.wan.only: true

   For an explanation of these connection options, refer to: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html.

Add a target node to the graph.

Your data target can be Amazon S3, or it can use information from an Amazon Glue Data Catalog or a connector to write data in a different location. For example, you can use a Data Catalog table to write to a database in Amazon RDS, or you can use a connector as your data target to write to data stores that are not natively supported in Amazon Glue.

If you choose a connector for your data target, you must choose a connection created for that connector. Also, if required by the connector provider, you must add options to provide additional information to the connector. If you use a connection that contains information for an Amazon secret, then you don’t need to provide the user name and password authentication in the connection options.

Optionally, add additional data sources and one or more transform nodes as described in Transform data with Amazon Glue managed transforms.

Configure the job properties as described in Modify the job properties, starting with step 3, and save the job.

Using the Amazon Glue Studio console, on the visual editor page, choose Run.

In the success banner, choose Run Details, or you can choose the Runs tab of the visual editor to view information about the job run.