IAM permissions required for KCL consumer applications - Amazon Kinesis Data Streams
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

IAM permissions required for KCL consumer applications

You must add the following permissions to the IAM role or user associated with your KCL consumer application.

Security best practices for Amazon dictate the use of fine-grained permissions to control access to different resources. Amazon Identity and Access Management (IAM) lets you manage users and user permissions in Amazon. An IAM policy explicitly lists actions that are allowed and the resources on which the actions are applicable.

The following table shows the minimum IAM permissions generally required for KCL consumer applications:

Minimum IAM permissions for KCL consumer applications
Service Actions Resources (ARNs) Purpose
Amazon Kinesis Data Streams

DescribeStream

DescribeStreamSummary

RegisterStreamConsumer

Kinesis data stream from which your KCL application will process the data.

arn:aws:kinesis:region:account:stream/StreamName

Before attempting to read records, the consumer checks if the data stream exists, if it's active, and if the shards are contained in the data stream.

Registers consumers to a shard.

Amazon Kinesis Data Streams

GetRecords

GetShardIterator

ListShards

Kinesis data stream from which your KCL application will process the data.

arn:aws:kinesis:region:account:stream/StreamName

Reads records from a shard.

Amazon Kinesis Data Streams

SubscribeToShard

DescribeStreamConsumer

Kinesis data stream from which your KCL application will process the data. Add this action only if you use enhanced fan-out (EFO) consumers.

arn:aws:kinesis:region:account:stream/StreamName/consumer/*

Subscribes to a shard for enhanced fan-out (EFO) consumers.

Amazon DynamoDB

CreateTable

DescribeTable

UpdateTable

Scan

GetItem

PutItem

UpdateItem

DeleteItem

Lease table (metadata table in DynamoDB created by KCL.

arn:aws:dynamodb:region:account:table/KCLApplicationName

These actions are required for KCL to manag the lease table created in DynamoDB.

Amazon DynamoDB

CreateTable

DescribeTable

Scan

GetItem

PutItem

UpdateItem

DeleteItem

Worker metrics and coordinator state table (metadata tables in DynamoDB) created by KCL.

arn:aws:dynamodb:region:account:table/KCLApplicationName-WorkerMetricStats

arn:aws:dynamodb:region:account:table/KCLApplicationName-CoordinatorState

Thess actions are required for KCL to manage the worker metrics and coordinator state metadata tables in DynamoDB.

Amazon DynamoDB

Query

Global secondary index on the lease table.

arn:aws:dynamodb:region:account:table/KCLApplicationName/index/*

This action is required for KCL to read the global secondary index of the lease table created in DynamoDB.

Amazon CloudWatch

PutMetricData

*

Upload metrics to CloudWatch that are useful for monitoring the application. The asterisk (*) is used because there is no spcific resource in CloudWatch on which the PutMetricData action is invoked.

Note

Replace "region," "account," "StreamName," and "KCLApplicationName" in the ARNs with your own Amazon Web Services Region, Amazon Web Services account number, Kinesis data stream name, and KCL application name respectively. KCL 3.x creates two more metadata tables in DynamoDB. For details about DynamoDB metadata tables created by KCL, see DynamoDB metadata tables and load balancing in KCL. If you use configurations to customize names of the metadata tables created by KCL, use those specified table names instead of KCL application name.

The following is an example policy document for a KCL consumer application.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "kinesis:DescribeStream", "kinesis:DescribeStreamSummary", "kinesis:RegisterStreamConsumer", "kinesis:GetRecords", "kinesis:GetShardIterator", "kinesis:ListShards" ], "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME" }, { "Effect": "Allow", "Action": [ "kinesis:SubscribeToShard", "kinesis:DescribeStreamConsumer" ], "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME/consumer/*" }, { "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:DescribeTable", "dynamodb:UpdateTable", "dynamodb:GetItem", "dynamodb:UpdateItem", "dynamodb:PutItem", "dynamodb:DeleteItem", "dynamodb:Scan" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME" ] }, { "Effect": "Allow", "Action": [ "dynamodb:CreateTable", "dynamodb:DescribeTable", "dynamodb:GetItem", "dynamodb:UpdateItem", "dynamodb:PutItem", "dynamodb:DeleteItem", "dynamodb:Scan" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME-WorkerMetricStats", "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME-CoordinatorState" ] }, { "Effect": "Allow", "Action": [ "dynamodb:Query" ], "Resource": [ "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/KCL_APPLICATION_NAME/index/*" ] }, { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData" ], "Resource": "*" } ] }

Before you use this example policy, check the following items:

  • Replace REGION with your Amazon Web Services Region (for example, us-east-1).

  • Replace ACCOUNT_ID with your Amazon Web Services account ID.

  • Replace STREAM_NAME with the name of your Kinesis data stream.

  • Replace CONSUMER_NAME with the name of your consumer, typically your application name when using KCL.

  • Replace KCL_APPLICATION_NAME with the name of your KCL application.