App Mesh observability troubleshooting

Make sure the App Mesh Virtual Node listener protocol is not set as TCP.

Make sure that the X-Ray container that is deployed with your application exposes UDP port 2000 and runs as user 1337. For more information, see the Amazon ECS X-Ray example on GitHub.

Make sure that the Envoy container has tracing enabled. If you are using the App Mesh Envoy image, you can enable X-Ray by setting the ENABLE_ENVOY_XRAY_TRACING environment variable to a value of 1 and the XRAY_DAEMON_PORT environment variable to 2000.

If you’ve instrumented X-Ray in your application code with one of the language-specific SDKs , then make sure that it is configured correctly by following the guides for your language.

If all of the previous items are configured correctly, then review the X-Ray container logs for errors and follow the guidance in Troubleshooting Amazon X-Ray. A more detailed explanation of X-Ray integration in App Mesh can be found in Integrating X-Ray with App Mesh.

Make sure that you are using the CloudWatch agent image for App Mesh. For more information, see App Mesh CloudWatch agent on GitHub.

Make sure that you have configured the CloudWatch agent for App Mesh appropriately by following the platform-specific usage instructions. For more information, see App Mesh CloudWatch agent on GitHub.

If all of the previous items are configured correctly, then review the CloudWatch agent container logs for errors and follow the guidance provided in Troubleshooting the CloudWatch agent.

To only set the sampling rate, use the XRAY_SAMPLING_RATE environment variable on the Envoy container. The value should be specified as a decimal between 0 and 1.00 (100%). For more information, see Amazon X-Ray variables.

To configure the localized custom sampling rules for the X-Ray tracer use the XRAY_SAMPLING_RULE_MANIFEST environment variable to specify a file path in the Envoy container file system. For more information, see Sampling rules in the Amazon X-Ray Developer Guide.

Use the ENVOY_TRACING_CFG_FILE environment variable to change your sampling rate. For more information, see Envoy configuration variables. Specify a custom tracing configuration and define local sampling rules. For more information, see Envoy X-Ray config.

Custom tracing configuration for the ENVOY_TRACING_CFG_FILE environment variable example:

tracing:
   http:
     name: envoy.tracers.xray
     typedConfig:
       "@type": type.googleapis.com/envoy.config.trace.v3.XRayConfig
       segmentName: foo/bar
       segmentFields:
         origin: AWS::AppMesh::Proxy
         aws:
           app_mesh:
             mesh_name: foo
             virtual_node_name: bar
       daemonEndpoint:
             protocol: UDP
             address: 127.0.0.1
             portValue: 2000
       samplingRuleManifest:
             filename: /tmp/sampling-rules.json

For details on configuration for the sampling rule manifest in the samplingRuleManifest property, see Configuring the X-Ray SDK for Go.