Storing logs - Amazon EMR
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Storing logs

To monitor your job progress on EMR Serverless and to troubleshoot failures, you can choose how EMR Serverless stores and serves application logs. When you submit a job run, you can specify managed storage, Amazon S3, or both as your logging option.

Managed storage

By default, EMR Serverless stores application logs securely in Amazon EMR managed storage for a maximum of 30 days.

Note

If you turn off the default option, Amazon EMR can't troubleshoot your jobs on your behalf.

To turn off this option from EMR Studio, deselect the Allow Amazon to retain logs for 30 days check box in the Additional settings section of the Submit job page.

To turn off this option from the Amazon CLI, use the managedPersistenceMonitoringConfiguration configuration when you submit a job run.

{ "monitoringConfiguration": { "managedPersistenceMonitoringConfiguration": { "enabled": false } } }

Amazon S3 buckets

Before your jobs can send log data to Amazon S3, you must include the following permissions in the permissions policy for the job runtime role. Replace DOC-EXAMPLE-BUCKET-LOGGING with the name of your logging bucket.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::DOC-EXAMPLE-BUCKET-LOGGING/*" ] } ] }

To set up an Amazon S3 bucket to store logs from the Amazon CLI, use the s3MonitoringConfiguration configuration when you start a job run. To do this, provide the following --configuration-overrides in the configuration.

{ "monitoringConfiguration": { "s3MonitoringConfiguration": { "logUri": "s3://DOC-EXAMPLE-BUCKET-LOGGING/logs/" } } }