Monitoring Amazon EMR events with CloudWatch
Amazon EMR tracks events and keeps information about them for up to seven days in the Amazon EMR console. Amazon EMR records events when there is a change in the state of clusters, instance groups, instance fleets, automatic scaling policies, or steps. Events capture the date and time the event occurred, details about the affected elements, and other critical data points.
The following table lists Amazon EMR events, along with the state or state change that the event indicates, the severity of the event, event type, event code, and event messages. Amazon EMR represents events as JSON objects and automatically sends them to an event stream. The JSON object is important when you set up rules for event processing using CloudWatch Events because rules seek to match patterns in the JSON object. For more information, see Events and event patterns and Amazon EMR events in the Amazon CloudWatch Events User Guide.
Note
To ensure that we provide you with the most pertinent information, we continuously refine our error messages. For that reason, we recommend that you don’t parse the text from the messages to initiate next actions in your workflow.
Cluster start events
| State or state change | Severity | Event type | Event code | Message |
|---|---|---|---|---|
CREATING |
WARN |
Amazon EMR instance fleet provisioning | EC2 provisioning - Insufficient Instance Capacity | We are not able to create your Amazon EMR cluster ClusterId
(ClusterName) for Instance Fleet InstanceFleetID
Amazon EC2 has insufficient Spot capacity for Instance type
[Instancetype1, Instancetype2] and insufficient
On-Demand capacity for Instance type [Instancetype3,
Instancetype4] in Availability Zone [AvailabilityZone1,
AvaliabilityZone2]. Check here documentation
for more information on how to respond to this event. |
CREATING |
WARN |
Amazon EMR instance group provisioning | EC2 provisioning - Insufficient Instance Capacity | We are not able to create your Amazon EMR cluster ClusterId
(ClusterName) for Instance Group InstancegroupID
Amazon EC2 has insufficient [Spot or On-Demand] capacity for
Instance type Instancetype in Availability Zone
AvailabilityZone. Check here documentation
for more information on how to respond to this event. |
STARTING
|
INFO
|
EMR cluster state change |
none |
Amazon EMR cluster |
STARTING
|
INFO
|
EMR cluster state change |
none |
NoteApplies only to clusters with the instance fleets configuration and multiple Availability Zones selected within Amazon EC2. Amazon EMR cluster |
STARTING
|
INFO
|
EMR cluster state change |
none |
Amazon EMR cluster |
WAITING
|
INFO
|
EMR cluster state change |
none |
Amazon EMR cluster - or - Amazon EMR cluster NoteA cluster in the |
Note
The events with event code EC2 provisioning - Insufficient Instance
Capacity periodically emit when your EMR cluster encounters an
insufficient capacity error from Amazon EC2 for your instance fleet or instance group
during cluster creation or resize operation. For information on how to respond to
these events, see Responding to Amazon EMR cluster
insufficient instance capacity events.
Cluster termination events
| State or state change | Severity | Event type | Event code | Message |
|---|---|---|---|---|
TERMINATED
|
The severity depends on the reason for the state change, as shown in the following:
|
EMR cluster state change |
none |
Amazon EMR Cluster |
TERMINATED_WITH_ERRORS
|
CRITICAL
|
EMR cluster state change |
none |
Amazon EMR Cluster |
Instance fleet state-change events
Note
The instance fleets configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.0 and 5.0.3.
| State or state change | Severity | Event type | Event code | Message |
|---|---|---|---|---|
From |
INFO
|
none | Provisioning for instance fleet |
|
From |
INFO
|
none | A resize for instance fleet |
|
From |
INFO
|
none | The resizing operation for instance fleet
|
|
From |
INFO
|
none | The resizing operation for instance fleet
|
|
SUSPENDED
|
ERROR
|
none | Instance fleet |
|
RESIZING
|
WARNING
|
none | The resizing operation for instance fleet
|
|
|
INFO
|
none | The resizing operation for instance fleet
|
|
|
INFO
|
none | A resizing operation for instance fleet
|
Instance fleet resize events
| Event type | Severity | Event code | Message |
|---|---|---|---|
Amazon EMR instance fleet resize |
ERROR |
Spot Provisioning timeout |
The Resize operation for Instance Fleet
|
Amazon EMR instance fleet resize |
ERROR |
On-Demand Provisioning timeout |
The Resize operation for Instance Fleet
|
Amazon EMR instance fleet resize |
WARNING |
EC2 provisioning - Insufficient Instance Capacity | We are not able to complete the resize operation for Instance
Fleet |
Amazon EMR instance fleet resize |
WARNING |
Spot Provisioning Timeout - Continuing Resize |
We're still provisioning Spot capacity for the Instance Fleet
resize operation that initiated at |
Amazon EMR instance fleet resize |
WARNING |
On-Demand Provisioning Timeout - Continuing Resize |
We're still provisioning On-Demand capacity for the Instance
Fleet resize operation that initiated at |
Note
The provisioning timeout events are emitted when Amazon EMR stops provisioning Spot or On-demand capacity for the fleet after the timeout expires. For information on how to respond to these events, see Responding to Amazon EMR cluster instance fleet resize timeout events .
Instance group events
| Event type | Severity | Event code | Message |
|---|---|---|---|
From |
INFO
|
none | The resizing operation for instance group
|
From |
INFO
|
none | A resize for instance group |
SUSPENDED
|
ERROR
|
none | Instance group |
RESIZING
|
WARNING
|
none | The resizing operation for instance group
|
Amazon EMR instance group resize |
WARNING |
EC2 provisioning - Insufficient Instance Capacity | We are not able to complete the resize operation that started
at |
From |
INFO
|
none | A resize for instance group |
Note
With Amazon EMR version 5.21.0 and later, you can override cluster configurations and specify additional configuration classifications for each instance group in a running cluster. You do this by using the Amazon EMR console, the Amazon Command Line Interface (Amazon CLI), or the Amazon SDK. For more information, see Supplying a Configuration for an Instance Group in a Running Cluster.
The following table lists Amazon EMR events for the reconfiguration operation, along with the state or state change that the event indicates, the severity of the event, and event messages.
| State or state change | Severity | Message |
|---|---|---|
RUNNING
|
INFO
|
A reconfiguration for instance group
|
From |
INFO
|
The reconfiguration operation for instance group
|
From |
INFO
|
A reconfiguration for instance group
|
RESIZING
|
INFO
|
Reconfiguring operation towards configuration version
|
RECONFIGURING
|
INFO
|
Resizing operation towards instance count Num for
instance group InstanceGroupID in the Amazon EMR cluster
ClusterId (ClusterName) is temporarily blocked at
Time because the instance group is in
State. |
RECONFIGURING
|
WARNING
|
The reconfiguration operation for instance group
|
RECONFIGURING
|
INFO
|
Configurations are reverting to the previous successful version
number |
From |
INFO
|
Configurations were successfully reverted to the previous
successful version |
From |
CRITICAL
|
Failed to revert to the previous successful version
|
Automatic scaling policy events
| State or state change | Severity | Message |
|---|---|---|
PENDING
|
INFO
|
An Auto Scaling policy was added to instance group
- or - The Auto Scaling policy for instance group
|
ATTACHED
|
INFO
|
The Auto Scaling policy for instance group
|
|
INFO
|
The Auto Scaling policy for instance group
|
FAILED
|
ERROR
|
The Auto Scaling policy for instance group
- or - The Auto Scaling policy for instance group
|
Step events
| State or state change | Severity | Message |
|---|---|---|
PENDING
|
INFO
|
Step |
CANCEL_PENDING
|
WARN
|
Step |
RUNNING
|
INFO
|
Step |
COMPLETED
|
INFO
|
Step |
CANCELLED
|
WARN
|
Cancellation request has succeeded for cluster step
|
FAILED
|
ERROR
|
Step |
Viewing events with the Amazon EMR console
For each cluster, you can view a simple list of events in the details pane, which lists events in descending order of occurrence. You can also view all events for all clusters in a region in descending order of occurrence.
If you don't want a user to see all cluster events for a region, add a statement that
denies permission ("Effect": "Deny") for the
elasticmapreduce:ViewEventsFromAllClustersInConsole action to a policy
that is attached to the user.
Note
We’ve redesigned the Amazon EMR console to make it easier to use. See What's new with the console? to learn about the differences between the old and new console experiences.