Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Customizing the log location for step log files

Beginning in Amazon EMR version 7.11, you can now customize the S3 logging behavior for steps on a per-step basis. For a single step, you can choose a unique S3 bucket where the step's logs are uploaded. You can also choose a unique KMS key which is used to encrypt the step's logs at rest, on S3. These options take precedence over any cluster-wide logging settings that are configured when launching the cluster.

Prerequisites

Step log configuration

When you submit a step to EMR, you can configure the step's logging behavior via StepMonitoringConfiguration. The StepMonitoringConfiguration contains the S3MonitoringConfiguration object where you can specify an S3 logging bucket and/or a KMS key for the step.

The following example shows you how you can customize a step's S3 bucket and KMS key from a python script:

import boto3

emr_client = boto3.client("emr", region_name="us-east-1")

# Define your step:
example_step = [
    {
        "Name": "Example Step for StepMonitoringConfiguration",
        "ActionOnFailure": "CONTINUE",
        "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": ["bash", "-c", "echo 1"]
        },
        "StepMonitoringConfiguration": {
            "S3MonitoringConfiguration": {
                "LogUri": "s3://your-s3-bucket/", # Replace this with your S3 bucket
                "EncryptionKeyArn": "arn:aws:kms:your-kms-key-arn" # Replace this with your KMS key ARN
            }
        }
    }
]

response = emr_client.add_job_flow_steps(
    JobFlowId="j-xxxxxxxxxxxxx",  # Replace this with your EMR cluster ID
    Steps=example_step
)

Considerations