Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting Amazon Glue for Ray errors from logs

Amazon Glue provides access to logs that are emitted by Ray processes during the job run. If you encounter errors or unexpected behavior in Ray jobs, first gather information from the logs to determine the cause of failure. We also provide similar logs for interactive sessions. Sessions logs are provided with the /aws-glue/ray/sessions prefix.

Log lines are sent to CloudWatch in real time, as your job is run. Print statements are appended to the CloudWatch logs after the run completes. Logs are retained for two weeks after a job is run.

Inspecting Ray job logs

When a job fails, gather your job name and job run ID. You can find these in the Amazon Glue console. Navigate to the job page, and then navigate to the Runs tab. Ray job logs are stored in the following dedicated CloudWatch log groups.

Troubleshooting Ray job errors

To understand the organization of Ray log groups, and to find the log groups that will help you troubleshoot your errors, it helps to have background information about Ray architecture.

In Amazon Glue ETL, a worker corresponds to an instance. When you configure workers for an Amazon Glue job, you're setting the type and quantity of instances that are dedicated to the job. Ray uses the term worker in different ways.

Ray uses head node and worker node to distinguish the responsibilities of an instance within a Ray cluster. A Ray worker node can host multiple actor processes that perform computations to achieve the result of your distributed computation. Actors that run a replica of a function are called replicas. Replica actors can also be called worker processes. Replicas can also run on the head node, which is known as the head because it runs additional processes to coordinate the cluster.

Each actor that contributes to your computation generates its own log stream. This provides us with some insights:

For more information about Ray architecture, see Architecture Whitepapers in the Ray documentation.

Problem area: Amazon S3 access

Check the failure message of the job run. If that doesn't provide enough information, check /aws-glue/ray/jobs/script-log/.

Problem area: PIP dependency management

Check /aws-glue/ray/jobs/ray-runtime-env-log/.

Problem area: Inspecting intermediate values in main process

Write to stderr or stdout from your main script, and retrieve logs from /aws-glue/ray/jobs/script-log/.

Problem area: Inspecting intermediate values in a child process

Write to stderr or stdout from your remote function. Then, retrieve logs from /aws-glue/ray/jobs/ray-worker-out-logs/ or /aws-glue/ray/jobs/ray-worker-err-logs/. Your function might have run on any replica, so you might have to examine multiple logs to find your intended output.

Problem area: Interpreting IP addresses in error messages

In certain error situations, your job might emit an error message that contains an IP address. These IP addresses are ephemeral, and are used by the cluster to identify and communicate between nodes. Logs for a node will be published to a log stream with a unique suffix based on the IP address.

In CloudWatch, you can filter down your logs to inspect those specific to this IP address by identifying this suffix. For example, given FAILED_IP and JOB_RUN_ID, you can identify the suffix with:

filter @logStream like /JOB_RUN_ID/
| filter @message like /IP-/
| parse @message "IP-[*]" as ip
| filter ip like /FAILED_IP/
| fields replace(ip, ":", "_") as uIP
| stats count_distinct by uIP as logStreamSuffix
| display logStreamSuffix