IAM permissions required for KCL consumer applications
You must add the following permissions to the IAM role or user associated with your KCL consumer application.
Security best practices for Amazon dictate the use of fine-grained permissions to control access to different resources. Amazon Identity and Access Management (IAM) lets you manage users and user permissions in Amazon. An IAM policy explicitly lists actions that are allowed and the resources on which the actions are applicable.
The following table shows the minimum IAM permissions generally required for KCL consumer applications:
Service | Actions | Resources (ARNs) | Purpose |
---|---|---|---|
Amazon Kinesis Data Streams |
|
Kinesis data stream from which your KCL application will process the data.
|
Before attempting to read records, the consumer checks if the data stream exists, if it's active, and if the shards are contained in the data stream. Registers consumers to a shard. |
Amazon Kinesis Data Streams |
|
Kinesis data stream from which your KCL application will process the data.
|
Reads records from a shard. |
Amazon Kinesis Data Streams |
|
Kinesis data stream from which your KCL application will process the data. Add this action only if you use enhanced fan-out (EFO) consumers.
|
Subscribes to a shard for enhanced fan-out (EFO) consumers. |
Amazon DynamoDB |
|
Lease table (metadata table in DynamoDB created by KCL.
|
These actions are required for KCL to manag the lease table created in DynamoDB. |
Amazon DynamoDB |
|
Worker metrics and coordinator state table (metadata tables in DynamoDB) created by KCL.
|
Thess actions are required for KCL to manage the worker metrics and coordinator state metadata tables in DynamoDB. |
Amazon DynamoDB |
|
Global secondary index on the lease table.
|
This action is required for KCL to read the global secondary index of the lease table created in DynamoDB. |
Amazon CloudWatch |
|
* |
Upload metrics to CloudWatch that are useful for monitoring the
application. The asterisk (*) is used because there is no spcific
resource in CloudWatch on which the |
Note
Replace "region," "account," "StreamName," and "KCLApplicationName" in the ARNs with your own Amazon Web Services Region, Amazon Web Services account number, Kinesis data stream name, and KCL application name respectively. KCL 3.x creates two more metadata tables in DynamoDB. For details about DynamoDB metadata tables created by KCL, see DynamoDB metadata tables and load balancing in KCL. If you use configurations to customize names of the metadata tables created by KCL, use those specified table names instead of KCL application name.
The following is an example policy document for a KCL consumer application.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:DescribeStreamSummary", "kinesis:RegisterStreamConsumer", "kinesis:GetRecords", "kinesis:GetShardIterator", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME" }, { "Effect": "Allow", "Action": [ "kinesis:SubscribeToShard", "kinesis:DescribeStreamConsumer" ], "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME/consumer/*" }, { "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:DescribeTable", "dynamodb:UpdateTable", "dynamodb:GetItem", "dynamodb:UpdateItem", "dynamodb:PutItem", "dynamodb:DeleteItem", "dynamodb:Scan" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME" ] }, { "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:DescribeTable", "dynamodb:GetItem", "dynamodb:UpdateItem", "dynamodb:PutItem", "dynamodb:DeleteItem", "dynamodb:Scan" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME-WorkerMetricStats", "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME-CoordinatorState" ] }, { "Effect": "Allow", "Action": [ "dynamodb:Query" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME/index/*" ] }, { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": "*" } ] }
Before you use this example policy, check the following items:
-
Replace REGION with your Amazon Web Services Region (for example, us-east-1).
-
Replace ACCOUNT_ID with your Amazon Web Services account ID.
-
Replace STREAM_NAME with the name of your Kinesis data stream.
-
Replace CONSUMER_NAME with the name of your consumer, typically your application name when using KCL.
-
Replace KCL_APPLICATION_NAME with the name of your KCL application.