Continuous logging for Amazon Glue jobs - Amazon Glue
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Continuous logging for Amazon Glue jobs

Amazon Glue provides real-time, continuous logging for Amazon Glue jobs. You can view real-time Apache Spark job logs in Amazon CloudWatch, including driver logs, executor logs, and an Apache Spark job progress bar. Viewing real-time logs provides you with a better perspective on the running job.

When you start an Amazon Glue job, it sends the real-time logging information to CloudWatch (every 5 seconds and before each executor termination) after the Spark application starts running. You can view the logs on the Amazon Glue console or the CloudWatch console dashboard.

The continuous logging feature includes the following capabilities:

  • Continuous logging with a default filter to reduce high verbosity in the logs

  • Continuous logging with no filter

  • A custom script logger to log application-specific messages

  • A console progress bar to track the running status of the current Amazon Glue job

For information about how continuous logging is supported in Amazon Glue version 2.0, see Running Spark ETL Jobs with Reduced Startup Times.

You can restrict access to CloudWatch log groups or streams for IAM roles to read the logs. For more details on restricting access, see Using identity-based policies (IAM policies) for CloudWatch Logs in the CloudWatch documentation.

Note

You may incur additional charges when you enable continuous logging and additional CloudWatch log events are created. For more information, see Amazon CloudWatch pricing .

Topics