Managing a data source - Amazon OpenSearch Service
Monitoring with CloudWatch metrics data sourcesEnabling and disabling data sources Monitoring with Amazon BudgetDeleting a data source
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Managing a data source

Managing your data source is an important part of maintaining the reliability, availability, and performance of direct query data sources and your other Amazon solutions. Amazon provides the following tools to monitor, report when something is wrong, and take automatic actions when appropriate.

Monitoring with CloudWatch metrics data sources

You can monitor direct query using CloudWatch. CloudWatch collects raw data and processes it into readable, near real-time metrics. These statistics are kept for 15 months, so that you can access historical information and gain a better perspective on how your web application or service is performing.

You can also set alarms to monitor certain thresholds, and send notifications or take actions when those thresholds are met. For more information, see What is Amazon CloudWatch.

Direct query reports the following metrics:

Metric Description
AsyncQueryCreateAPI

The total number of requests made to the API for creating asynchronous queries.

Relevant statistics:

Average, Maximum, Sum

Dimensions: ClientId,DomainName

Frequency: 60 seconds

AsyncQueryGetApiRequestCount

The total number of requests made to the API for retrieving asynchronous query results.

Relevant statistics:

Average, Maximum, Sum

Dimensions: ClientId, DomainName

Frequency: 60 seconds

AsyncQueryCancelApiRequestCount

The total number of requests made to the API for canceling asynchronous queries.

Relevant statistics:

Average, Maximum, Sum

Dimensions: ClientId, DomainName

Frequency: 60 seconds

AsyncQueryGetApiFailedRequestCusErrCount

The number of failed requests when retrieving asynchronous query results due to customer-related errors (e.g., invalid query ID).

Relevant statistics:

Average, Maximum, Sum

Dimensions: ClientId, DomainName

Frequency: 60 seconds

AsyncQueryCancelApiFailedRequestCusErrCount

The number of failed requests when retrieving asynchronous query results due to customer-related errors (e.g., invalid query ID).

Relevant statistics: Average,Maximum, Sum

Dimensions: ClientId,DomainName

Frequency: 60 seconds

AsyncQueryCancelApiFailedRequestSysErrCount

The number of failed requests when creating asynchronous queries due to customer-related errors.

Relevant statistics: Average, Maximum, Sum

Dimensions: ClientId, DomainName

Frequency: 60 seconds

AsyncQueryGetApiFailedRequestSysErrCount

The number of failed requests when retrieving asynchronous query results due to system-related errors.

Relevant statistics: Average, Maximum, Sum

Dimensions: ClientId, DomainName

Frequency: 60 seconds

Enabling and disabling data sources

For circumstances when you want to halt direct query usage for a data source, you can opt to disable the data source. Disabling a data source will finish executing existing queries and halt all new queries from being executed by the user.

Accelerations setup to boost query performance such as skipping indexes, materialized views, covering index will be set to manual once a data source is disabled. Once a data source is set to active after being disabled, user queries will run as expected. Accelerations which were previously setup and set to manual, will need to be manually configured to run on a schedule again.

Monitoring with Amazon Budget

Amazon OpenSearch Service is populating OCU usage data at the account level into Billing and Cost Management's Cost Explorer. Customers can account for OCU usage at the account level and set thresholds and alerts when thresholds have been crossed.

The format of the usage type to filter on in Cost Explorer looks like RegionCode-DirectQueryOCU (OCU-hours). Customers wanting to be notified when DirectQueryOCU (OCU-Hours) usage meets their threshold, can create an Amazon Budgets account and configure an alert based on the threshold they set. Optionally, customers can elect to setup an Amazon SNS topic, which will turn off a data source in the event a threshold criteria is met.

Note

Usage data in Amazon Budgets is not real-time and may be delayed up to 8 hours.

Deleting an Amazon OpenSearch Service data source with Amazon S3

When you delete a data source, Amazon OpenSearch Service removes it from your domain. OpenSearch Service also removes indexes associated with the data source. Your transactional data isn't deleted from Amazon S3, but Amazon S3 doesn't send new data to OpenSearch Service.

You can delete a data source integration using the Amazon Web Services Management Console or the OpenSearch Service API.

To delete a data source

  1. Navigate to the Amazon OpenSearch Service console at https://console.aws.amazon.com/aos/.

  2. From the left navigation pane, choose Domains.

  3. Select the domain that you want to delete a data source for. This opens the domain details page. Choose the Connections tab below the general information and find the Direct query section.

  4. Select the data source you want to delete, choose Delete, and confirm deletion.

Use the DeleteDataSource API operation to delete an existing data souce in your domain.

POST https://es.region.amazonaws.com/2021-01-01/opensearch/domain/domain-name/dataSource/data-source-name