Troubleshooting CloudWatch Events
Amazon EventBridge is the preferred way to manage your events. CloudWatch Events and EventBridge are the same underlying service and API, but EventBridge provides more features. Changes you make in either CloudWatch or EventBridge will appear in each console. For more information, see Amazon EventBridge. |
You can use the steps in this section to troubleshoot CloudWatch Events.
Topics
- My rule was triggered but my Lambda function was not invoked
- I have just created/modified a rule but it did not match a test event
- My rule did not self-trigger at the time specified in the ScheduleExpression
- My rule did not trigger at the time that I expected
- My rule matches IAM API calls but my rule was not triggered
- My rule is not working because the IAM role associated with the rule is ignored when the rule is triggered
- I created a rule with an EventPattern that is supposed to match a resource, but I don't see any events that match the rule
- My event's delivery to the target experienced a delay
- Some events were never delivered to my target
- My rule was triggered more than once in response to one event. What guarantee does CloudWatch Events offer for triggering rules or delivering events to the targets?
- Preventing Infinite Loops
- My events are not delivered to the target Amazon SQS queue
- My rule is being triggered but I don't see any messages published into my Amazon SNS topic
- My Amazon SNS topic still has permissions for CloudWatch Events even after I deleted the rule associated with the Amazon SNS topic
- Which IAM condition keys can I use with CloudWatch Events
- How can I tell when CloudWatch Events rules are broken
My rule was triggered but my Lambda function was not invoked
Make sure you have the right permissions set for your Lambda function. Run the following command using Amazon CLI (replace the function name with your function and use the Amazon Region your function is in):
aws lambda get-policy --function-name
MyFunction
--regionus-east-1
You should see an output similar to the following:
{ "Policy": "{\"Version\":\"2012-10-17\", \"Statement\":[ {\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:123456789012:rule/MyRule\"}}, \"Action\":\"lambda:InvokeFunction\", \"Resource\":\"arn:aws:lambda:us-east-1:123456789012:function:MyFunction\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Sid\":\"MyId\"} ], \"Id\":\"default\"}" }
If you see the following:
A client error (ResourceNotFoundException) occurred when calling the GetPolicy operation: The resource you requested does not exist.
Or, you see the output but you can't locate events.amazonaws.com as a trusted entity in the policy, run the following command:
aws lambda add-permission \ --function-name
MyFunction
\ --statement-idMyId
\ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012
:rule/MyRule
If the policy is incorrect, you can also edit the rule in the CloudWatch Events console by removing and then adding it back to the rule. The CloudWatch Events console will set the correct permissions on the target.
If you're using a specific Lambda alias or version, you must add the
--qualifier
parameter in the aws lambda get-policy
and
aws lambda add-permission
commands.
aws lambda add-permission \ --function-name
MyFunction
\ --statement-idMyId
\ --action 'lambda:InvokeFunction' \ --principal events.amazonaws.com \ --source-arn arn:aws:events:us-east-1:123456789012
:rule/MyRule
--qualifieralias or version
Another reason the Lambda function would fail to trigger is if the policy you see when running get-policy
contains a
SourceAccount
field. A
SourceAccount
setting prevents CloudWatch Events from being able to invoke the function.
I have just created/modified a rule but it did not match a test event
When you make a change to a rule or to its targets, incoming events might not
immediately start or stop matching to new or updated rules. Allow a short period of time
for changes to take effect. If, after this short period, events still do not match, you
can also check CloudWatch metrics for your rule such as
TriggeredRules
, Invocations
, and
FailedInvocations
for further debugging. For more information about these
metrics, see Amazon CloudWatch Events Metrics and Dimensions in the Amazon CloudWatch User Guide.
If the rule is triggered by an event from an Amazon service, you can also use the TestEventPattern
action to test the event pattern of
your rule with a test event to make sure the event pattern of your rule is correctly
set. For more information, see TestEventPattern in the Amazon CloudWatch Events API Reference.
My rule did not self-trigger at the time specified in the ScheduleExpression
ScheduleExpressions are in UTC. Make sure you have set the schedule for rule to self-trigger in the UTC timezone. If the ScheduleExpression is correct, then follow the steps under I have just created/modified a rule but it did not match a test event.
My rule did not trigger at the time that I expected
CloudWatch Events doesn't support setting an exact start time when you create a rule to run every time period. The count down to run time begins as soon as you create the rule.
You can use a cron expression to invoke targets at a specified time. For example, you
can use a cron expression to create a rule that is triggered every 4 hours exactly on 0
minute. In the CloudWatch console, you'd use the cron expression 0 0/4 * * ? *
,
and with the Amazon CLI you'd use the cron expression cron(0 0/4 * * ? *)
. For
example, to create a rule named TestRule that is triggered every 4 hours using the
Amazon CLI, you would type the following at a command prompt:
aws events put-rule --name
TestRule
--schedule-expression 'cron(0 0/4 * * ? *)'
You can use the 0/5 * * * ? *
cron expression to trigger a rule every 5
minutes. For example:
aws events put-rule --name
TestRule
--schedule-expression 'cron(0/5 * * * ? *)'
CloudWatch Events does not provide second-level precision in schedule expressions. The finest resolution using a cron expression is a minute. Due to the distributed nature of the CloudWatch Events and the target services, the delay between the time the scheduled rule is triggered and the time the target service honors the execution of the target resource might be several seconds. Your scheduled rule will be triggered within that minute but not on the precise 0th second.
My rule matches IAM API calls but my rule was not triggered
The IAM service is only available in the US East (N. Virginia) Region, so any Amazon API call events from IAM are only available in that region. For more information, see CloudWatch Events Event Examples From Supported Services.
My rule is not working because the IAM role associated with the rule is ignored when the rule is triggered
IAM roles for rules are only used for relating events to Kinesis streams. For Lambda functions and Amazon SNS topics, you need to provide resource-based permissions.
Make sure your regional Amazon STS endpoints are enabled. CloudWatch Events talks to the regional Amazon STS endpoints when assuming the IAM role you provided. For more information, see Activating and Deactivating Amazon STS in an Amazon Region in the IAM User Guide.
I created a rule with an EventPattern that is supposed to match a resource, but I don't see any events that match the rule
Most services in Amazon treat the colon (:) or forward slash (/) as the same character in Amazon Resource Names (ARNs). However, CloudWatch Events uses an exact match in event patterns and rules. Be sure to use the correct ARN characters when creating event patterns so that they match the ARN syntax in the event to match.
Moreover, not every event has the resources field populated (such as Amazon API call events from CloudTrail).
My event's delivery to the target experienced a delay
CloudWatch Events tries to deliver an event to a target for up to 24 hours, except in
scenarios where your target resource is constrained. The first attempt
is made as soon as the event arrives in the event stream. However, if the target service
is having problems, CloudWatch Events automatically reschedules
another delivery in the future. If 24 hours has passed since the arrival of event, no
more attempts are scheduled and the FailedInvocations
metric is published
in CloudWatch. We recommend that you create a CloudWatch alarm on the FailedInvocations
metric.
Some events were never delivered to my target
If a target of a CloudWatch Events rule is constrained for a prolonged time, CloudWatch Events may not retry delivery. For example, if the target is not provisioned to handle the incoming event traffic and the target service is throttling the requests that CloudWatch Events makes on your behalf, then CloudWatch Events may not retry delivery.
My rule was triggered more than once in response to one event. What guarantee does CloudWatch Events offer for triggering rules or delivering events to the targets?
In rare cases, the same rule can be triggered more than once for a single event or scheduled time, or the same target can be invoked more than once for a given triggered rule.
Preventing Infinite Loops
In CloudWatch Events, it is possible to create rules that lead to infinite loops, where a rule is fired repeatedly. For example, a rule might detect that ACLs have changed on an S3 bucket, and trigger software to change them to the desired state. If the rule is not written carefully, the subsequent change to the ACLs fires the rule again, creating an infinite loop.
To prevent this, write the rules so that the triggered actions do not re-fire the same rule. For example, your rule could fire only if ACLs are found to be in a bad state, instead of after any change.
An infinite loop can quickly cause higher than expected charges. We recommend that you use budgeting, which alerts you when charges exceed your specified quota. For more information, see Managing Your Costs with Budgets.
My events are not delivered to the target Amazon SQS queue
The Amazon SQS queue may be encrypted. If you create a rule with an encrypted Amazon SQS queue as a target, you must have the following section included in your KMS key policy for the event to be successfully delivered to the encrypted queue.
{ "Sid": "Allow CWE to use the key", "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource": "*" }
My rule is being triggered but I don't see any messages published into my Amazon SNS topic
Make sure you have the right permission set for your Amazon SNS topic. Run the following command using Amazon CLI (replace the topic ARN with your topic and use the Amazon Region your topic is in):
aws sns get-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic"
You should see policy attributes similar to the following:
"{\"Version\":\"2012-10-17\", \"Id\":\"__default_policy_ID\", \"Statement\":[{\"Sid\":\"__default_statement_ID\", \"Effect\":\"Allow\", \"Principal\":{\"AWS\":\"*\"}, \"Action\":[\"SNS:Subscribe\", \"SNS:ListSubscriptionsByTopic\", \"SNS:DeleteTopic\", \"SNS:GetTopicAttributes\", \"SNS:Publish\", \"SNS:RemovePermission\", \"SNS:AddPermission\", \"SNS:Receive\", \"SNS:SetTopicAttributes\"], \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\", \"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}},{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\", \"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}]}"
If you see a policy similar to the following, you have only the default policy set:
"{\"Version\":\"2008-10-17\", \"Id\":\"__default_policy_ID\", \"Statement\":[{\"Sid\":\"__default_statement_ID\", \"Effect\":\"Allow\", \"Principal\":{\"AWS\":\"*\"}, \"Action\":[\"SNS:Subscribe\", \"SNS:ListSubscriptionsByTopic\", \"SNS:DeleteTopic\", \"SNS:GetTopicAttributes\", \"SNS:Publish\", \"SNS:RemovePermission\", \"SNS:AddPermission\", \"SNS:Receive\", \"SNS:SetTopicAttributes\"], \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\", \"Condition\":{\"StringEquals\":{\"AWS:SourceOwner\":\"123456789012\"}}}]}"
If you don't see events.amazonaws.com
with Publish permission in your
policy, use the Amazon CLI to set topic policy attribute.
Copy the current policy and add the following statement to the list of statements:
{\"Sid\":\"Allow_Publish_Events\", \"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"}, \"Action\":\"sns:Publish\", \"Resource\":\"arn:aws:sns:us-east-1:123456789012:MyTopic\"}
The new policy should look like the one described earlier.
Set topic attributes with the Amazon CLI:
aws sns set-topic-attributes --region us-east-1 --topic-arn "arn:aws:sns:us-east-1:123456789012:MyTopic" --attribute-name Policy --attribute-value
NEW_POLICY_STRING
If the policy is incorrect, you can also edit the rule in the CloudWatch Events console by removing and then adding it back to the rule. CloudWatch Events sets the correct permissions on the target.
My Amazon SNS topic still has permissions for CloudWatch Events even after I deleted the rule associated with the Amazon SNS topic
When you create a rule with Amazon SNS as the target, CloudWatch Events adds the permission to your Amazon SNS topic on your behalf. If you delete the rule shortly after you create it, CloudWatch Events might be unable to remove the permission from your Amazon SNS topic. If this happens, you can remove the permission from the topic using the Amazon SNS set-topic-attributes command.
Which IAM condition keys can I use with CloudWatch Events
CloudWatch Events supports the Amazon-wide condition keys (see Available Keys in the IAM User Guide), plus the following service-specific condition keys.
How can I tell when CloudWatch Events rules are broken
You can use the following alarm to notify you when your CloudWatch Events rules are broken.
To create an alarm to alert when rules are broken
-
Open the CloudWatch console at https://console.amazonaws.cn/cloudwatch/
. -
Choose Create Alarm. In the CloudWatch Metrics by Category pane, choose Events Metrics.
-
In the list of metrics, select FailedInvocations.
-
Above the graph, choose Statistic, Sum.
-
For Period, choose a value, for example 5 minutes. Choose Next.
-
Under Alarm Threshold, for Name, type a unique name for the alarm, for example myFailedRules. For Description, type a description of the alarm, for example Rules are not delivering events to targets.
-
For is, choose >= and 1. For for, enter 10.
-
Under Actions, for Whenever this alarm, choose State is ALARM.
-
For Send notification to, select an existing Amazon SNS topic or create a new one. To create a new topic, choose New list. Type a name for the new Amazon SNS topic, for example: myFailedRules.
-
For Email list, type a comma-separated list of email addresses to be notified when the alarm changes to the ALARM state.
-
Choose Create Alarm.