Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Managing data retention for NextGen time series collections

With a NextGen time series collection, you control how long OpenSearch Serverless retains data in an index before it deletes the data. You set this retention period with the index.data.retention index setting. You can use this setting to control storage costs and to meet data compliance requirements.

Only NextGen time series collections support the index.data.retention setting. Search and vector search collections don't.

Key differences from data lifecycle policies

Data retention for NextGen collections differs from data lifecycle policies in the following ways:

Using the data retention index setting

For NextGen time series collections, you configure data retention directly on indexes with the PUT <index>/_settings API and the index.data.retention setting. This setting replaces the lifecycle policy mechanism that Classic collections use.

The index.data.retention setting has the following properties.

API usage patterns for data retention

The following REST API requests show common ways to set and remove the index.data.retention setting. In each request, replace my-index with the name of the index that you want to configure.

The following REST API request sets retention on one or more indexes.

PUT my-index/_settings
{
  "index.data.retention": "7"
}

A successful request returns the following response.

{
  "acknowledged": true
}

The following REST API request sets retention on all indexes in a collection with a wildcard.

PUT */_settings
{
  "index.data.retention": "7"
}

The following REST API request sets retention on indexes that match a prefix.

PUT error*/_settings
{
  "index.data.retention": "30"
}

The following REST API request sets retention on multiple specific indexes.

PUT access-2024,error-2024/_settings
{
  "index.data.retention": "7"
}

The following REST API request removes retention from an index with a null value.

PUT my-index/_settings
{
  "index.data.retention": null
}

The following REST API request sets retention when you create an index.

PUT my-index
{
  "settings": {
    "index.data.retention": "7"
  }
}

A successful request returns the following response.

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my-index"
}

Using index templates for retention

The PUT _settings API applies only to existing indexes. With an index template, you can automatically apply a retention setting to future indexes. When you create an index whose name matches the template pattern, the index inherits the retention setting.

The following REST API request creates an index template that applies a 30-day retention setting to new indexes whose names match the logs-* pattern.

PUT _index_template/my-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "index.data.retention": "30"
    }
  }
}

A successful request returns the following response.

{
  "acknowledged": true
}

If an index creation request specifies index.data.retention and a matching template also specifies it, the request value takes precedence over the template value.

Validation and error messages

OpenSearch Serverless validates the index.data.retention setting and returns an error when validation fails. The following list describes each validation scenario and the error that OpenSearch Serverless returns for it.

Java client usage

OpenSearch Serverless requires SigV4-signed requests. Use the OpenSearch Java client (opensearch-java) with AwsSdk2Transport, which signs each request. Set the signing service name to aoss.

The index.data.retention setting isn't available as a typed builder method, so set it with IndexSettings.Builder.customSettings. The following example creates an index named test-idx with a 5-day retention setting. Replace the endpoint and the Region with the values for your collection.

import org.opensearch.client.json.JsonData;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
import org.opensearch.client.opensearch.indices.IndexSettings;
import org.opensearch.client.transport.aws.AwsSdk2Transport;
import org.opensearch.client.transport.aws.AwsSdk2TransportOptions;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.regions.Region;

SdkHttpClient httpClient = ApacheHttpClient.builder().build();

OpenSearchClient client = new OpenSearchClient(
    new AwsSdk2Transport(
        httpClient,
        "your-collection-endpoint.us-east-1.aoss.amazonaws.com", // collection endpoint, without a scheme
        "aoss",                                                      // signing service name
        Region.US_EAST_1,
        AwsSdk2TransportOptions.builder().build()));

CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder()
    .index("test-idx")
    .settings(new IndexSettings.Builder()
        .customSettings("index.data.retention", JsonData.of("5"))
        .build())
    .build();

client.indices().create(createIndexRequest);

httpClient.close();

Related resources

For more information about related topics, see the following resources: