Migrating from X-Ray instrumentation to OpenTelemetry instrumentation

X-Ray is transitioning to OpenTelemetry (OTel) as its primary instrumentation standard for application tracing and observability. This strategic shift aligns Amazon with industry best practices and offers customers a more comprehensive, flexible, and future-ready solution for their observability needs. OpenTelemetry's wide adoption in the industry enables tracing of requests across diverse systems, including those outside Amazon that may not directly integrate with X-Ray.

This chapter provides recommendations for a smooth transition, and emphasizes the importance of migrating to OpenTelemetry-based solutions to ensure continued support and access to the latest features in application instrumentation and observability.

It is recommended to adopt OpenTelemetry as the observability solution for instrumenting your application.

Understanding OpenTelemetry

OpenTelemetry is an industry-standard observability framework that provides standardized protocols and tools for collecting telemetry data. It offers a unified approach to instrumenting, generating, collecting, and exporting telemetry data such as metrics, logs, and traces.

When you migrate from X-Ray SDKs to OpenTelemetry, you get the following benefits:

The OpenTelemetry collector provides more options for data collection formats and export destinations than the X-Ray daemon.

OpenTelemetry support in Amazon

Amazon provides multiple solutions for working with OpenTelemetry:

Using OpenTelemetry with Amazon CloudWatch

Amazon CloudWatch supports OpenTelemetry traces through client-side application instrumentation and native Amazon CloudWatch services such as Application Signals, Trace, Map, Metrics, and Logs. For more information, see OpenTelemetry.

Understanding OpenTelemetry concepts for migration

The following table maps X-Ray concepts to their OpenTelemetry equivalents. Understanding these mappings helps you translate your existing X-Ray instrumentation to OpenTelemetry:

Comparing features

The following table shows which features are supported in both services. Use this information to identify any gaps you need to address during migration:

Setting up and configuring tracing

To create traces in OpenTelemetry, you need a tracer. You get a tracer by initializing a Tracer Provider in your application. This is similar to how you use the X-Ray Recorder to configure X-Ray and create segments and subsegments in an X-Ray trace.

Understanding trace data structure

After understanding the basic concepts and feature mappings, you can learn about specific implementation details like trace data structure and sampling.

OpenTelemetry uses spans instead of segments and subsegments to structure trace data. Each span includes the following components:

When you migrate to OpenTelemetry, your spans are automatically converted to X-Ray segments or subsegments. This ensures your existing CloudWatch console experience remains unchanged.

Working with span attributes

The X-Ray SDK provides two ways to add data to segments and subsegments:

By default, OpenTelemetry span attributes are converted to metadata in X-Ray raw data. To convert specific attributes to annotations instead, add their keys to the aws.xray.annotations attributes list.

Detecting resources in your environment

OpenTelemetry uses Resource Detectors to collect metadata about the resources that generate telemetry data. This metadata is stored as Resource Attributes. For example, an entity producing telemetry could be an Amazon ECS cluster or an Amazon EC2 instance, and the Resource Attributes that can be recorded from these entities can include the Amazon ECS Cluster ARN or Amazon EC2 Instance ID.

Managing sampling strategies

Trace sampling helps you manage costs by collecting data from a representative subset of requests instead of all requests. Both OpenTelemetry and X-Ray support sampling, but implement it differently.

OpenTelemetry provides several built-in sampling strategies and lets you create custom ones. You can also configure an X-Ray Remote Sampler in some SDK languages to use X-Ray sampling rules with OpenTelemetry.

The additional sampling strategies from OpenTelemetry are:

For information about X-Ray sampling rules, see Sampling rules in the X-Ray console

For information about OpenTelemetry tail sampling, see Tail sampling processor

Managing trace context

X-Ray SDKs manage the Segment Context to correctly handle parent-child relationships between Segments and Subsegments in a trace. OpenTelemetry uses a similar mechanism to ensure that spans have the correct parent span. It stores and propagates tracing data throughout a request context. For example, when your application processes a request and creates a server span to represent that request, OpenTelemetry will store the server span in the OpenTelemetry Context so that when a child span is created, that child span can reference the span in the Context as its parent.

Propagating trace context

Both X-Ray and OpenTelemetry use HTTP headers to propagate trace context across services. This allows you to link trace data generated by different services and maintain sampling decisions.

The X-Ray SDK automatically propagates trace context using the X-Ray trace header. When one service calls another, the trace header contains the context needed to maintain parent-child relationships between traces.

OpenTelemetry supports multiple trace header formats for context propagation, including:

Configure and use the X-Ray Propagator to enable tracing across Amazon services. This allows you to propagate trace context to API Gateway endpoints and other services that support X-Ray.

Using library instrumentation

Both X-Ray and OpenTelemetry provide library instrumentation that requires minimal code changes to add tracing to your applications.

X-Ray provides library instrumentation functionalities. This allows you to add pre-built X-Ray instrumentations with minimal application code changes. These instrumentations support specific libraries like the Amazon SDK and HTTP Clients, as well as web frameworks like Spring Boot or Express.js.

OpenTelemetry's instrumentation libraries generate detailed spans for your libraries through library hooks or automatic code modification, requiring minimal code changes.

To determine if OpenTelemetry's Library Instrumentations supports your library, search for it in the OpenTelemetry Registry at OpenTelemetry Registry.

Exporting traces

X-Ray and OpenTelemetry use different methods to export trace data.

X-Ray trace export

The X-Ray SDKs use an emitter to send trace data:

OpenTelemetry trace export

OpenTelemetry uses configurable Span Exporters to send trace data:

Processing and forwarding traces

Both X-Ray and OpenTelemetry provide components to receive, process, and forward trace data.

X-Ray trace processing

The X-Ray Daemon handles trace processing:

OpenTelemetry trace processing

The OpenTelemetry Collector handles trace processing:

Span processing (OpenTelemetry-specific concept)

OpenTelemetry uses Span Processors to modify spans as they're created:

Baggage (OpenTelemetry-soecific concept)

OpenTelemetry's Baggage feature allows propagation of key-value data:

For information about the OpenTelemetry Collector, see OpenTelemetry Collector

For information about X-Ray concepts, see X-Ray concepts in the X-Ray Developer Guide

Migration overview

This section provides an overview of the code changes required for migration. The list below are language-specific guidance and X-Ray Daemon migration steps.

Recommendations for new and existing applications

For new and existing applications, it is recommended to use the following solutions to enable tracing in your applications:

After migrating to OpenTelemetry-based solutions, your CloudWatch experience will remain the same. You will still be able to view your traces in the same format in the CloudWatch console’s Traces and Trace Map pages, or retrieve your trace data through the X-Ray APIs.

Tracing setup changes

You need to replace the X-Ray setup with an OpenTelemetry setup.

Library instrumentation changes

Update your code to use OpenTelemetry Library Instrumentation instead of X-Ray Library Instrumentation for Amazon SDK, HTTP Clients, Web Frameworks, and other libraries. This generates OpenTelemetry Traces instead of X-Ray Traces.

Lambda environment instrumentation changes

To use OpenTelemetry in your Lambda functions, choose one of these setup options:

Manually creating trace data

Replace X-Ray segments and sub-segments with OpenTelemetry Spans:

To convert an attribute to an annotation, add its key to the aws.xray.annotations attribute list. For more information, see Enable Customized X-Ray Annotations.

Migrating from X-Ray Daemon to Amazon CloudWatch agent or OpenTelemetry collector

You can use either the CloudWatch agent or OpenTelemetry collector to receive traces from your instrumented applications and send them to X-Ray.

Migrating on Amazon EC2 or on-premises servers

Existing X-Ray Daemon setup

Installing the daemon

Your existing X-Ray Daemon usage was installed using one of these methods:

Configuring the daemon

Your existing X-Ray Daemon usage was configured using either:

./xray -c ~/xray-daemon.yaml

Running the daemon

Your existing X-Ray Daemon usage was started with the following command:

~/xray-daemon$ ./xray -o -n us-east-1

Removing the daemon

To remove the X-Ray Daemon from your Amazon EC2 instance:

Setting up the CloudWatch agent

Installing the agent

For installation instructions, see Installing the CloudWatch agent on an on-premises server.

Configuring the agent

Starting the agent

For instructions to start the agent, see Starting the CloudWatch agent using the command line.

Setting up the OpenTelemetry collector

Installing the collector

Download and install the OpenTelemetry collector for your operating system. For instructions, see Installing the collector.

Configuring the collector

Configure the following components in your collector:

extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000
  health_check:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  batch:

exporters:
  awsxray:
    region: 'us-east-1'

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [awsxray]
  extensions: [awsproxy, health_check]

Starting the collector

Run the collector with your configuration file:

otelcol --config=otel-collector-config.yaml

Migrating on Amazon ECS

Using the CloudWatch agent

For more information, see Deploying the CloudWatch agent to collect Amazon EC2 instance-level metrics on Amazon ECS .

Using the OpenTelemetry collector

Migrating on Elastic Beanstalk

Your existing X-Ray Daemon integration was turned on by using the Elastic Beanstalk console, or by configuring X-Ray Daemon in your application source code with a configuration file.

Using the CloudWatch agent

On the Amazon Linux 2 platform, configure the CloudWatch agent using an .ebextensions configuration file:

For more information about Elastic Beanstalk configuration files, see Advanced environment customization with configuration files.