Setting up Amazon CloudWatch alarms on Amazon Glue job profiles
Amazon Glue metrics are also available in Amazon CloudWatch. You can set up alarms on any Amazon Glue metric for scheduled jobs.
A few common scenarios for setting up alarms are as follows:
-
Jobs running out of memory (OOM): Set an alarm when the memory usage exceeds the normal average for either the driver or an executor for an Amazon Glue job.
-
Straggling executors: Set an alarm when the number of executors falls below a certain threshold for a large duration of time in an Amazon Glue job.
-
Data backlog or reprocessing: Compare the metrics from individual jobs in a workflow using a CloudWatch math expression. You can then trigger an alarm on the resulting expression value (such as the ratio of bytes written by a job and bytes read by a following job).
For detailed instructions on setting alarms, see Create
or Edit a CloudWatch Alarm
For monitoring and debugging scenarios using CloudWatch, see Job monitoring and debugging.