Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Multi-tier storage for Amazon OpenSearch Service

Multi-tier storage for Amazon OpenSearch Service is an intelligent data management solution that optimizes both performance and costs by managing data across different storage tiers. This architecture allows organizations to efficiently balance the trade-off between performance and cost by keeping frequently accessed data in high-performance hot storage while moving less frequently accessed data to more cost-effective warm storage.

Amazon OpenSearch Service offers two architecture options for hot/warm storage tier:

Multi-tier storage architecture

Key benefits

Prerequisites

Limitations

Things to note

Creating a multi-tier domain

Step 1: Create the domain

aws opensearch create-domain \
      --domain-name my-domain \
      --engine-version OpenSearch_3.3 \
      --cluster-config InstanceCount=3,InstanceType=or2.large.search,DedicatedMasterEnabled=true,DedicatedMasterType=m6g.large.search,DedicatedMasterCount=3,WarmEnabled=true,WarmCount=3,WarmType=oi2.2xlarge.search \
      --ebs-options EBSEnabled=true,VolumeType=gp2,VolumeSize=11 \
      --node-to-node-encryption-options Enabled=true \
      --encryption-at-rest-options Enabled=true \
      --domain-endpoint-options EnforceHTTPS=true,TLSSecurityPolicy=Policy-Min-TLS-1-2-2019-07 \
      --advanced-security-options '{"Enabled":true,"InternalUserDatabaseEnabled":true,"MasterUserOptions":{"MasterUserName":"user_name","MasterUserPassword":"your_pass"}}' \
      --access-policies '{"Version": "2012-10-17",		 	 	  "Statement":[{"Effect":"Allow","Principal":"*","Action":"es:*","Resource":"*"}]}' \
      --region us-east-1

Step 2: Verify warm nodes

aws opensearch describe-domain-nodes --domain-name my-domain --region us-east-1

Sample response (excerpt):

{
      "NodeType": "Warm",
      "InstanceType": "oi2.large.search",
      "NodeStatus": "Active"
    }

Managing tier migrations

Multi-tier domains support:

New tiering APIs

Migrate an index to warm:

curl -XPOST 'https://localhost:9200/index-name/_tier/warm'

Response:

{"acknowledged": true}

Migrate an index to hot:

curl -XPOST 'https://localhost:9200/index-name/_tier/hot'

Response:

{"acknowledged": true}

Check tiering status:

curl -XGET 'https://localhost:9200/index-name/_tier'

Example response:

{
      "tiering_status": {
         "index": "index-name",
         "state": "RUNNING_SHARD_RELOCATION",
         "source": "HOT",
         "target": "WARM",
         "start_time": 1745836500563,
         "shard_level_status": {
           "running": 0,
           "total": 100,
           "pending": 100,
           "succeeded": 0
         }
      }
    }

Detailed shard view:

curl 'https://localhost:9200/index1/_tier?detailed=true'

List all ongoing migrations (text):

curl 'https://localhost:9200/_tier/all'

List all ongoing migrations (JSON):

curl 'https://localhost:9200/_tier/all?format=json'

Filter by target tier:

curl 'https://localhost:9200/_tier/all?target=_warm'

Legacy UltraWarm APIs for compatibility

Migrate to warm:

curl -XPOST localhost:9200/_ultrawarm/migration/index2/_warm

Migrate to hot:

curl -XPOST localhost:9200/_ultrawarm/migration/index2/_hot

Check status:

curl -XGET localhost:9200/_ultrawarm/migration/index2/_status

Security configuration

If you enable multi-tier storage on a preexisting Amazon OpenSearch Service domain, the storage_tiering_manager role might not be defined on the domain. Non-admin users must be mapped to this role in order to manage warm indexes on domains using fine-grained access control. To manually create the storage_tiering_manager role, perform the following steps:

Best practices

As you implement multi-tier storage in your Amazon OpenSearch Service domains, consider the following best practices:

Monitoring metrics

Multi-tier storage domains provide additional metrics for monitoring warm tier performance. These metrics include both existing UltraWarm metrics and new metrics specific to the OpenSearch Optimized Instances architecture:

New metrics