View a markdown version of this page

Collect telemetry from compiled applications using eBPF - Amazon CloudWatch
Understanding how it worksDetermining when to use this approachCompleting the prerequisitesStep 1: Configure the CloudWatch agent to receive OTLP dataStep 2: Point the eBPF instrumentation at the agentStep 3: Verify your telemetry
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Collect telemetry from compiled applications using eBPF

You cannot instrument some applications by adding an OpenTelemetry SDK to the source. Examples include compiled binaries (such as Go, Rust, or C/C++), third-party or vendor binaries whose source you do not own, and applications whose libraries an in-process agent cannot attach to. For these applications, you can use eBPF-based auto-instrumentation to generate OpenTelemetry telemetry without changing application code. You then send that telemetry to CloudWatch through the CloudWatch agent.

Important

The CloudWatch agent is the OTLP receiver and exporter (the collection layer). It does not instrument your application. You pair the agent with a separate eBPF instrumentation tool that produces OTLP data. The open source OpenTelemetry project for this is OpenTelemetry eBPF Instrumentation (OBI) on the OpenTelemetry website. eBPF attaches probes at the kernel, network, and TLS layers. As a result, it captures HTTP, HTTPS, and gRPC traffic and produces rate, errors, and duration (RED) metrics and trace spans without per-library code changes. For the languages, runtime versions, and libraries that OBI supports, see the OBI documentation. Coverage might vary by language and library, including which client and server spans it produces. Validate the specific frameworks your application uses before you rely on this approach.

Understanding how it works

The telemetry flows through the following pipeline.

  1. eBPF auto-instrumentation generates telemetry without a code change and emits OTLP.

  2. The CloudWatch agent, which is built on the OpenTelemetry Collector, receives the OTLP data. For more information, see Amazon CloudWatch agent.

  3. The agent forwards traces to Amazon X-Ray (Transaction Search) and metrics to CloudWatch.

Determining when to use this approach

Consider this approach in the following cases:

  • You have compiled binaries that you cannot add an SDK to.

  • You have vendor or third-party code that you do not own.

  • You must keep an existing third-party monitoring agent in place.

  • You want RED metrics and traces without a code change or a redeploy of the application.

Completing the prerequisites

Before you begin, complete the following prerequisites:

  • Install CloudWatch agent version 1.300070.0 or later. Earlier versions do not support the agent's OTLP receiver. On Amazon EKS, use the CloudWatch Observability EKS add-on version 6.6.0 or later so that the agent Service exposes the OTLP ports.

  • Attach the CloudWatchAgentServerPolicy managed policy. The agent signs requests to the CloudWatch OTLP endpoints with its own credentials.

  • Enable Transaction Search so that spans reach the X-Ray OTLP endpoint. For more information, see Transaction Search.

  • The eBPF instrumentation tool runs on Linux and requires a kernel with eBPF support and a set of Linux capabilities (such as CAP_BPF). The exact capabilities depend on the instrumentation tool and which features you enable, so grant the least privilege the tool documents rather than running it fully privileged. See the OBI documentation.

  • If the agent runs in a subnet without internet egress, make sure it can reach every endpoint it exports to. These endpoints are CloudWatch for metrics and X-Ray for traces (which Transaction Search uses). If you use the logs path, the agent must also reach CloudWatch Logs. In a private network, you typically provide this reachability with interface VPC endpoints (Amazon PrivateLink). For the CloudWatch interface VPC endpoints, see Using CloudWatch, CloudWatch Synthetics, and CloudWatch Network Monitoring with interface VPC endpoints. For the X-Ray VPC endpoint, see Using X-Ray with interface VPC endpoints.

Step 1: Configure the CloudWatch agent to receive OTLP data

Add an opentelemetry section to your CloudWatch agent configuration file and include the otlp source under collect. Bind the receiver to 127.0.0.1 so that the agent accepts OTLP only from the local host. In containerized deployments where producers run outside the agent's own container network—for example an Amazon ECS sidecar, or the Amazon EKS add-on's cloudwatch-agent Service—set the endpoints to 0.0.0.0 instead. The task, pod, or Service boundary then controls what can reach the ports.

{
  "opentelemetry": {
    "collect": {
      "otlp": {
        "grpc_endpoint": "127.0.0.1:4317",
        "http_endpoint": "127.0.0.1:4318"
      }
    }
  }
}

The agent forwards traces to X-Ray and metrics to CloudWatch automatically. For the full agent OTLP configuration, see Amazon CloudWatch agent.

Run one OTLP receiver per host

Run a single OTLP receiver—the CloudWatch agent—for each host network namespace, and have every telemetry producer (your eBPF instrumentation and any OpenTelemetry SDKs) export to it. Only one process can bind the default OTLP ports (4317 and 4318) in a given network namespace, so a second collector that also uses the defaults fails to start. If you previously ran a separate OpenTelemetry or ADOT collector, remove it when you move collection to the agent. The eBPF instrumentation is a producer that exports OTLP, so it does not open the OTLP receiver ports. On Amazon EKS, each pod has its own network namespace, so this conflict occurs only when a pod uses host networking or a host port. If two receivers must coexist, the ports are configurable—set grpc_endpoint and http_endpoint to non-default values.

Step 2: Point the eBPF instrumentation at the agent

Configure the eBPF tool's OTLP exporter to send to the local agent, using gRPC on port 4317 or HTTP on port 4318 (for example, http://localhost:4318). The following deployment notes apply:

  • Amazon EC2—Run the agent and the eBPF tool on the same instance, and export OTLP to localhost.

  • Amazon EKS—The CloudWatch Observability EKS add-on runs the agent and exposes ports 4317 and 4318 on the cloudwatch-agent Service. Run the eBPF instrumentation as a DaemonSet and point its exporter at that Service.

For the eBPF tool's own configuration, see the OBI documentation.

Step 3: Verify your telemetry

Verify that your telemetry is arriving:

  • Metrics—Open the CloudWatch console, go to Query Studio, and run a PromQL query for your metric. OpenTelemetry metric names keep their dots in CloudWatch Query Studio, so query by the metric's OpenTelemetry name with the __name__ matcher rather than a Prometheus-style normalized name—for example, {__name__="http.server.request.duration"} (the actual name depends on what your instrumentation emits).

  • Traces and spans—Use Transaction Search. For more information, see Transaction Search.

Running alongside an existing monitoring agent

eBPF-based instrumentation runs out of process and does not inject into the application process. As a result, it can typically run alongside an existing in-process monitoring agent that does not instrument the same compiled runtime. Because eBPF tools require elevated privileges and some monitoring agents also use eBPF, validate resource usage and probe behavior in a test environment before you deploy to production.