OpenSearch Service connections - Amazon Glue
Configuring OpenSearch ServiceRead from OpenSearch ServiceWrite to OpenSearch ServiceOpenSearch Service connection option reference
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

OpenSearch Service connections

You can use Amazon Glue for Spark to read from and write to tables in OpenSearch Service in Amazon Glue 4.0 and later versions. You can define what to read from OpenSearch Service with an OpenSearch query. You connect to OpenSearch Service using HTTP basic authentication credentials stored in Amazon Secrets Manager through a Amazon Glue connection. This feature is not compatible with OpenSearch Service serverless.

For more information about Amazon OpenSearch Service, see the Amazon OpenSearch Service documentation.

Configuring OpenSearch Service connections

To connect to OpenSearch Service from Amazon Glue, you will need to create and store your OpenSearch Service credentials in a Amazon Secrets Manager secret, then associate that secret with a OpenSearch Service Amazon Glue connection.

Prerequisites:

  • Identify the domain endpoint, aosEndpoint and port, aosPort you would like to read from, or create the resource by following instructions in the Amazon OpenSearch Service documentation. For more information on creating a domain, see Creating and managing Amazon OpenSearch Service domains in the Amazon OpenSearch Service documentation.

    An Amazon OpenSearch Service domain endpoint will have the following default form, https://search-domainName-unstructuredIdContent.region.es.amazonaws.com. For more information on identifying your domain endpoint, see Creating and managing Amazon OpenSearch Service domains in the Amazon OpenSearch Service documentation.

    Identify or generate HTTP basic authentication credentials, aosUser and aosPassword for your domain.

To configure a connection to OpenSearch Service:

  1. In Amazon Secrets Manager, create a secret using your OpenSearch Service credentials. To create a secret in Secrets Manager, follow the tutorial available in Create an Amazon Secrets Manager secret in the Amazon Secrets Manager documentation. After creating the secret, keep the Secret name, secretName for the next step.

    • When selecting Key/value pairs, create a pair for the key opensearch.net.http.auth.user with the value aosUser.

    • When selecting Key/value pairs, create a pair for the key opensearch.net.http.auth.pass with the value aosPassword.

  2. In the Amazon Glue console, create a connection by following the steps in Adding an Amazon Glue connection. After creating the connection, keep the connection name, connectionName, for future use in Amazon Glue.

    • When selecting a Connection type, select OpenSearch Service.

    • When selecting a Domain endpoint, provide aosEndpoint.

    • When selecting a port, provide aosPort.

    • When selecting an Amazon Secret, provide secretName.

After creating a Amazon Glue OpenSearch Service connection, you will need to perform the following steps before running your Amazon Glue job:

  • Grant the IAM role associated with your Amazon Glue job permission to read secretName.

  • In your Amazon Glue job configuration, provide connectionName as an Additional network connection.

Reading from OpenSearch Service indexes

Prerequisites:

  • A OpenSearch Service index you would like to read from, aosIndex.

  • A Amazon Glue OpenSearch Service connection configured to provide auth and network location information. To acquire this, complete the steps in the previous procedure, To configure a connection to OpenSearch Service. You will need the name of the Amazon Glue connection, connectionName.

This example reads an index from Amazon OpenSearch Service. You will need to provide the pushdown parameter.

For example: 

opensearch_read = glueContext.create_dynamic_frame.from_options(
    connection_type="opensearch",
    connection_options={
        "connectionName": "connectionName",
        "opensearch.resource": "aosIndex",
        "pushdown": "true",
    }
)

You can also provide a query string to filter the results returned in your DynamicFrame. You will need to configure opensearch.query.

opensearch.query can take a URL query parameter string queryString or a query DSL JSON object queryObject. For more information about the query DSL, see Query DSL in the OpenSearch documentation. To provide a URL query parameter string, prepend ?q= to your query, as you would in a fully qualified URL. To provide a query DSL object, string escape the JSON object before providing it.

For example: 


            queryObject = "{ "query": { "multi_match": { "query": "Sample", "fields": [ "sample" ] } } }"
            queryString = "?q=queryString"
            
            opensearch_read_query = glueContext.create_dynamic_frame.from_options(
    connection_type="opensearch",
    connection_options={
        "connectionName": "connectionName",
        "opensearch.resource": "aosIndex",
        "opensearch.query": queryString,
        "pushdown": "true",
    }
)

For more information about how to build a query outside of its specific syntax, see Query string syntax in the OpenSearch documentation.

When reading from OpenSearch collections that contain array type data, you must specify which fields are array type in your method call using the opensearch.read.field.as.array.include parameter.

For example, when reading the following document, you will encounter the genre and actor array fields:

{
    "_index": "movies",
    "_id": "2",
    "_version": 1,
    "_seq_no": 0,
    "_primary_term": 1,
    "found": true,
    "_source": {
        "director": "Frankenheimer, John",
        "genre": [
            "Drama",
            "Mystery",
            "Thriller",
            "Crime"
        ],
        "year": 1962,
        "actor": [
            "Lansbury, Angela",
            "Sinatra, Frank",
            "Leigh, Janet",
            "Harvey, Laurence",
            "Silva, Henry",
            "Frees, Paul",
            "Gregory, James",
            "Bissell, Whit",
            "McGiver, John",
            "Parrish, Leslie",
            "Edwards, James",
            "Flowers, Bess",
            "Dhiegh, Khigh",
            "Payne, Julie",
            "Kleeb, Helen",
            "Gray, Joe",
            "Nalder, Reggie",
            "Stevens, Bert",
            "Masters, Michael",
            "Lowell, Tom"
        ],
        "title": "The Manchurian Candidate"
    }
}

In this case, you would include those field names in your method call. For example:

"opensearch.read.field.as.array.include": "genre,actor"

If your array field is nested inside of your document structure, refer to it using dot notation: "genre,actor,foo.bar.baz". This would specify an array baz included in your source document through the embedded document foo containing the embedded document bar.

Writing to OpenSearch Service tables

This example writes information from an existing DynamicFrame, dynamicFrame to OpenSearch Service. If the index already has information, Amazon Glue will append data from your DynamicFrame. You will need to provide the pushdown parameter.

Prerequisites:

  • A OpenSearch Service table you would like to write to. You will need identification information for the table. Let's call this tableName.

  • A Amazon Glue OpenSearch Service connection configured to provide auth and network location information. To acquire this, complete the steps in the previous procedure, To configure a connection to OpenSearch Service. You will need the name of the Amazon Glue connection, connectionName.

For example: 

glueContext.write_dynamic_frame.from_options(
    frame=dynamicFrame,
    connection_type="opensearch",
    connection_options={
      "connectionName": "connectionName",
      "opensearch.resource": "aosIndex",
    },
)

OpenSearch Service connection option reference

  • connectionName — Required. Used for Read/Write. The name of a Amazon Glue OpenSearch Service connection configured to provide auth and network location information to your connection method.

  • opensearch.resource — Required. Used for Read/Write. Valid Values: OpenSearch index names. The name of the index your connection method will interact with.

  • opensearch.query — Used for Read. Valid Values: String escaped JSON or, when this string begins with ?, the search part of a URL. An OpenSearch query that filters what should be retrieved when reading. For more information on using this parameter, consult the previous section Reading from OpenSearch Service indexes.

  • pushdown — Required if. Used for Read. Valid Values: boolean. Instructs Spark to pass read queries down to OpenSearch so the database only returns relevant documents.

  • opensearch.read.field.as.array.include — Required if reading array type data. Used for Read. Valid Values: comma separated lists of field names. Specifies fields to read as arrays from OpenSearch documents. For more information on using this parameter, consult the previous section Reading from OpenSearch Service indexes.