Trace Analytics with Amazon OpenSearch Ingestion - Amazon OpenSearch Service
OpenTelemetry trace sourceProcessorsOpenSearch sinkPipeline configuration
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Trace Analytics with Amazon OpenSearch Ingestion

You can use Amazon OpenSearch Ingestion to collect OpenTelemetry trace data and transform it for use in OpenSearch Service. The following example pipeline uses three sub-pipelines to monitor Trace Analytics: entry-pipeline, span-pipeline, and service-map-pipeline.

OpenTelemetry trace source

The Otel trace source plugin accepts trace data from the OpenTelemetry Collector. The plugin follows the OpenTelemetry Protocol and officially supports industry-standard encryption HTTPS.

Processors

You can use the following processors for Trace Analytics:

  • OTel trace – Receives a collection of span records from the source and performs stateful processing, extraction, and completion of fields.

  • OTel trace group – Fills in missing trace group fields in the collection of span records.

  • Service-map – Performs preprocessing for trace data and builds metadata to display service-map dashboards.

OpenSearch sink

The OpenSearch sink plugin provides indexes and index templates that are specific to Trace Analytics. The following OpenSearch indexes are specific to Trace Analytics:

  • otel-v1-apm-span – Stores the output from the OTel trace processor.

  • otel-v1-apm-service-map – Stores the output from the Service-map processor.

Pipeline configuration

The following example pipeline supports Observability for OpenSearch Dashboards. The first sub-pipeline (entry-pipeline) receives data from the OpenTelemetry Collector and uses two other sub-pipelines as sinks.

The span-pipeline sub-pipeline parses the trace data and enriches and ingests the span documents into a span index. The service-map-pipeline sub-pipeline aggregates traces into a service map and writes documents to a service map index.

version: "2"
entry-pipeline:
  source:
    otel_trace_source:
      # Provide the path for ingestion. This will be the endpoint URI path in the OpenTelemetry Exporter configuration.
      # ${pipelineName} will be replaced with the sub-pipeline name. In this case it would be "/entry-pipeline/v1/traces". 
      path: "/${pipelineName}/v1/traces"
  processor:
    - trace_peer_forwarder
  sink:
    - pipeline:
        name: "span-pipeline"
    - pipeline:
        name: "service-map-pipeline"

span-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - otel_traces
  sink:
    - opensearch:
        ...
        index_type: trace-analytics-raw

service-map-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  processor:
    - service_map
  sink:
    - opensearch:
        ...
        index_type: trace-analytics-service-map

You must run the OpenTelemetry Collector in your environment to send data to the ingestion endpoint. For another example pipeline, see the Trace Analytics pipeline blueprint. For more information, see Using blueprints to create a pipeline.