Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Ingest pipelines in Amazon OpenSearch Serverless

An ingest pipeline is an ordered sequence of processors that transforms and enriches documents before they are indexed. For example, a pipeline can rename a field, normalize the case of a string, parse a timestamp, or generate vector embeddings for semantic search.

You define a pipeline once, and then apply it to incoming documents either as the default pipeline for an index or on individual indexing requests. Because the pipeline runs in Amazon OpenSearch Serverless, your documents receive the same transformations regardless of which client or integration sends them, which reduces the need for custom preprocessing logic in your applications.

With OpenSearch Serverless, you can use a subset of the ingest pipeline functionality that's available in OpenSearch. This topic describes the processors that OpenSearch Serverless supports, the limitations compared to the open-source implementation, and how to create and manage pipelines. For complete conceptual documentation on ingest pipelines, see Ingest pipelines on the OpenSearch Documentation website.

Supported processors

With OpenSearch Serverless, you can use a curated set of ingest processors. OpenSearch Serverless rejects pipelines that reference any other processor type when you create or update the pipeline.

Machine learning processors

The following processors support neural search and semantic enrichment workflows. They require a configured machine learning model connection. For more information, see Configure Neural Search and Hybrid Search on OpenSearch Serverless.

Data transformation processors

The following processors perform general-purpose transformations on document fields.

For configuration details for each processor, see Ingest processors on the OpenSearch Documentation website.

Limitations

Ingest pipelines in OpenSearch Serverless have the following limitations compared to the open-source OpenSearch implementation:

Pricing

Ingest pipelines run on the compute that processes indexing requests for your collection. The processing time that your pipelines consume counts toward the indexing OpenSearch Compute Units (OCUs) for your collection, in the same way as other indexing activity. Pipelines with more processors, or with compute-intensive processors, consume more indexing capacity than simple pipelines. For more information about how indexing capacity scales and is billed, see Managing capacity limits for Amazon OpenSearch Serverless and Amazon OpenSearch Service pricing.

Machine learning processors such as text_embedding additionally invoke the remote model that you configured, and the model provider bills model inference separately.

Required permissions

Ingest pipeline operations map to the following OpenSearch Serverless data access policy permissions. The permissions apply to the collection resource type.

The following example data access policy grants a principal permission to manage pipelines in a collection. Replace collection-name with the name of your collection, account-id with your Amazon account ID, and role-name with the name of the IAM role that manages pipelines.

[
    {
        "Description": "Manage pipelines",
        "Rules": [
            {
                "ResourceType": "collection",
                "Resource": ["collection/collection-name"],
                "Permission": [
                    "aoss:CreateCollectionItems",
                    "aoss:DescribeCollectionItems",
                    "aoss:DeleteCollectionItems"
                ]
            }
        ],
        "Principal": [
            "arn:aws-cn:iam::account-id:role/role-name"
        ]
    }
]

For the complete mapping of OpenSearch API operations to OpenSearch Serverless permissions, see Supported operations and plugins in Amazon OpenSearch Serverless.

Working with ingest pipelines

You create and manage ingest pipelines with the OpenSearch REST API against your collection endpoint. All requests must be signed with Amazon Signature Version 4 using the service name aoss. For more information, see Signing HTTP requests with other clients.

Create or update a pipeline

To create a pipeline, send a PUT request to the _ingest/pipeline endpoint with the pipeline name and an ordered list of processors. Documents pass through the processors in the order that they're defined. To update an existing pipeline, send the same request with the modified definition.

The following example creates a pipeline that trims a field, lowercases it, and renames it. Replace my-pipeline with the name of your pipeline.

PUT _ingest/pipeline/my-pipeline
{
  "description": "Normalizes the category field",
  "processors": [
    {
      "trim": {
        "field": "category"
      }
    },
    {
      "lowercase": {
        "field": "category"
      }
    },
    {
      "rename": {
        "field": "category",
        "target_field": "category_normalized"
      }
    }
  ]
}

The request returns the following response:

{
  "acknowledged": true
}

The following example creates a pipeline that chunks a text field and generates vector embeddings for each chunk. It requires a configured model. For more information, see Configure Neural Search and Hybrid Search on OpenSearch Serverless.

PUT _ingest/pipeline/nlp-ingest-pipeline
{
  "description": "Chunks text and generates embeddings",
  "processors": [
    {
      "text_chunking": {
        "algorithm": {
          "fixed_token_length": {
            "token_limit": 384
          }
        },
        "field_map": {
          "text-field": "text-chunks-field"
        }
      }
    },
    {
      "text_embedding": {
        "model_id": "my-model-id",
        "field_map": {
          "text-chunks-field": "embedding-field"
        }
      }
    }
  ]
}

For more information, see Create pipeline on the OpenSearch Documentation website.

Get a pipeline

To retrieve the definition of a pipeline, send a GET request with the pipeline name:

GET _ingest/pipeline/my-pipeline

The request returns the pipeline definition:

{
  "my-pipeline": {
    "description": "Normalizes the category field",
    "processors": [
      {
        "trim": {
          "field": "category"
        }
      },
      {
        "lowercase": {
          "field": "category"
        }
      },
      {
        "rename": {
          "field": "category",
          "target_field": "category_normalized"
        }
      }
    ]
  }
}

For more information, see Get pipeline on the OpenSearch Documentation website.

List pipelines

To list all of the pipelines in a collection, send a GET request without a pipeline name:

GET _ingest/pipeline

The request returns the definitions of all of the pipelines in the collection, keyed by pipeline name.

Simulate a pipeline

Before you apply a pipeline to live traffic, you can simulate it against sample documents and inspect the transformed output. To simulate an existing pipeline, send a POST request with the pipeline name and the sample documents:

POST _ingest/pipeline/my-pipeline/_simulate
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "1",
      "_source": {
        "category": "  Home Appliances  "
      }
    }
  ]
}

The response shows each document as it would be indexed after the pipeline runs. In this example, the pipeline trimmed and lowercased the category field and renamed it to category_normalized:

{
  "docs": [
    {
      "doc": {
        "_index": "my-index",
        "_id": "1",
        "_source": {
          "category_normalized": "home appliances"
        },
        "_ingest": {
          "timestamp": "2026-10-28T16:23:40.895265415Z"
        }
      }
    }
  ]
}

You can also simulate a pipeline definition that you haven't created yet by including the definition in the request body:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "lowercase": {
          "field": "category"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "my-index",
      "_id": "1",
      "_source": {
        "category": "Home Appliances"
      }
    }
  ]
}

For more information, see Simulate pipeline on the OpenSearch Documentation website.

Delete a pipeline

To delete a pipeline that you no longer need, send a DELETE request with the pipeline name:

DELETE _ingest/pipeline/my-pipeline

The request returns the following response:

{
  "acknowledged": true
}

Apply a pipeline to indexing requests

To run a pipeline on every document that's indexed into an index, set the index.default_pipeline setting when you create the index:

PUT my-index
{
  "settings": {
    "index.default_pipeline": "my-pipeline"
  }
}

The request returns the following response:

{
  "acknowledged": true,
  "shards_acknowledged": false,
  "index": "my-index"
}

Alternatively, specify the pipeline on an individual indexing or bulk request with the pipeline query parameter:

POST my-index/_doc?pipeline=my-pipeline
{
  "category": "  Home Appliances  "
}

The response confirms that the document was indexed:

{
  "_index": "my-index",
  "_id": "93VTg6EBtHTIBjFym5yW",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 0,
    "successful": 0,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 0
}