Clusters - Amazon Managed Streaming for Apache Kafka
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Clusters

Use this resource to create an Amazon MSK cluster and to get a list of existing clusters.

URI

/v1/clusters

HTTP methods

GET

Operation ID: ListClusters

Returns a list of all the MSK clusters.

Query parameters
NameTypeRequiredDescription
nextTokenStringFalse

The paginated results marker. When the result of the operation is truncated, the call returns NextToken in the response. To get the next batch, provide this token in your next request.

clusterNameFilterStringFalse

Specify a prefix of the name of the clusters that you want to list. The service lists all the clusters whose names start with this prefix.

maxResultsStringFalse

The maximum number of results to return in the response. If there are more results, the response includes a NextToken parameter.

Responses
Status codeResponse modelDescription
200 ListClustersResponse

Successful response.

400Error

The request isn't valid because the input is incorrect. Correct your input and then submit it again.

401Error

The request is not authorized. The provided credentials couldn't be validated.

403Error

Access forbidden. Check your credentials and then retry your request.

404Error

The resource could not be found due to incorrect input. Correct the input, then retry the request.

429Error

429 response

500Error

There was an unexpected internal server error. Retrying your request might resolve the issue.

503Error

503 response

POST

Operation ID: CreateCluster

Creates a new MSK cluster. The following Python 3.6 examples shows how you can create a cluster that's distributed over two Availability Zones. Before you run this Python script, replace the example subnet and security-group IDs with the IDs of your subnets and security group. When you create an MSK cluster, its brokers get evenly distributed over a number of Availability Zones that's equal to the number of subnets that you specify in the BrokerNodeGroupInfo parameter. In this example, you can add a third subnet to get a cluster that's distributed over three Availability Zones.

import boto3 client = boto3.client('kafka') response = client.create_cluster( BrokerNodeGroupInfo={ 'BrokerAZDistribution': 'DEFAULT', 'ClientSubnets': [ 'subnet-012345678901fedcba', 'subnet-9876543210abcdef01' ], 'InstanceType': 'kafka.m5.large', 'SecurityGroups': [ 'sg-012345abcdef789789' ] }, ClusterName='SalesCluster', EncryptionInfo={ 'EncryptionInTransit': { 'ClientBroker': 'TLS_PLAINTEXT', 'InCluster': True } }, EnhancedMonitoring='PER_TOPIC_PER_BROKER', KafkaVersion='2.2.1', NumberOfBrokerNodes=2 ) print(response)
Responses
Status codeResponse modelDescription
200 CreateClusterResponse

Successful response.

400Error

The request isn't valid because the input is incorrect. Correct your input and then submit it again.

401Error

The request is not authorized. The provided credentials couldn't be validated.

403Error

Access forbidden. Check your credentials and then retry your request.

404Error

The resource could not be found due to incorrect input. Correct the input, then retry the request.

409Error

This cluster name already exists. Retry your request using another name.

429Error

429 response

500Error

There was an unexpected internal server error. Retrying your request might resolve the issue.

503Error

503 response

OPTIONS

Enable CORS by returning the correct headers.

Responses
Status codeResponse modelDescription
200None

The default response for a CORS method.

Schemas

Request bodies

{ "encryptionInfo": { "encryptionInTransit": { "inCluster": boolean, "clientBroker": enum }, "encryptionAtRest": { "dataVolumeKMSKeyId": "string" } }, "numberOfBrokerNodes": integer, "configurationInfo": { "arn": "string", "revision": integer }, "enhancedMonitoring": enum, "storageMode": enum, "clusterName": "string", "kafkaVersion": "string", "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "unauthenticated": { "enabled": boolean }, "tls": { "certificateAuthorityArnList": [ "string" ], "enabled": boolean } }, "loggingInfo": { "brokerLogs": { "s3": { "bucket": "string", "prefix": "string", "enabled": boolean }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "cloudWatchLogs": { "logGroup": "string", "enabled": boolean } } }, "brokerNodeGroupInfo": { "clientSubnets": [ "string" ], "instanceType": "string", "connectivityInfo": { "publicAccess": { "type": "string" } }, "securityGroups": [ "string" ], "brokerAZDistribution": enum, "storageInfo": { "ebsStorageInfo": { "volumeSize": integer } } }, "openMonitoring": { "prometheus": { "nodeExporter": { "enabledInBroker": boolean }, "jmxExporter": { "enabledInBroker": boolean } } }, "tags": { } }

Response bodies

{ "nextToken": "string", "clusterInfoList": [ { "encryptionInfo": { "encryptionInTransit": { "inCluster": boolean, "clientBroker": enum }, "encryptionAtRest": { "dataVolumeKMSKeyId": "string" } }, "zookeeperConnectString": "string", "creationTime": "string", "zookeeperConnectStringTls": "string", "loggingInfo": { "brokerLogs": { "s3": { "bucket": "string", "prefix": "string", "enabled": boolean }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "cloudWatchLogs": { "logGroup": "string", "enabled": boolean } } }, "currentVersion": "string", "tags": { }, "numberOfBrokerNodes": integer, "clusterArn": "string", "activeOperationArn": "string", "enhancedMonitoring": enum, "storageMode": enum, "clusterName": "string", "stateInfo": { "code": "string", "message": "string" }, "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "unauthenticated": { "enabled": boolean }, "tls": { "certificateAuthorityArnList": [ "string" ], "enabled": boolean } }, "state": enum, "brokerNodeGroupInfo": { "clientSubnets": [ "string" ], "instanceType": "string", "connectivityInfo": { "publicAccess": { "type": "string" } }, "securityGroups": [ "string" ], "brokerAZDistribution": enum, "storageInfo": { "ebsStorageInfo": { "volumeSize": integer } } }, "openMonitoring": { "prometheus": { "nodeExporter": { "enabledInBroker": boolean }, "jmxExporter": { "enabledInBroker": boolean } } }, "currentBrokerSoftwareInfo": { "configurationRevision": integer, "kafkaVersion": "string", "configurationArn": "string" } } ] }
{ "clusterArn": "string", "clusterName": "string", "state": enum }
{ "message": "string", "invalidParameter": "string" }

Properties

BrokerAZDistribution

This parameter is currently not in use.

  • DEFAULT

BrokerLogs

The broker logs configuration for this MSK cluster.

PropertyTypeRequiredDescription
cloudWatchLogs

CloudWatchLogs

False

Details of the CloudWatch Logs destination for broker logs.

firehose

Firehose

False

Details of the Kinesis Data Firehose delivery stream that is the destination for broker logs.

s3

S3

False

Details of the Amazon S3 destination for broker logs.

BrokerNodeGroupInfo

Describes the setup to be used for the broker nodes in the cluster.

PropertyTypeRequiredDescription
brokerAZDistribution

BrokerAZDistribution

False

This parameter is currently not in use.

clientSubnets

Array of type string

True

The list of subnets to connect to in the client virtual private cloud (VPC). Amazon creates elastic network interfaces inside these subnets. Client applications use elastic network interfaces to produce and consume data.

If you use the US West (N. California) Region, specify exactly two subnets. For other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in distinct Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly across the subnets that you specify.

Client subnets can't occupy the Availability Zone with ID use1-az3.

connectivityInfo

ConnectivityInfo

False

Information about the cluster's connectivity setting.

instanceType

string

MinLength: 5

MaxLength: 32

True

The type of Amazon EC2 instances to use for brokers. The following instance types are allowed: kafka.m5.large, kafka.m5.xlarge, kafka.m5.2xlarge, kafka.m5.4xlarge, kafka.m5.8xlarge, kafka.m5.12xlarge, kafka.m5.16xlarge, and kafka.m5.24xlarge.

securityGroups

Array of type string

False

The security groups to associate with the elastic network interfaces in order to specify who can connect to and communicate with the Amazon MSK cluster. If you don't specify a security group, Amazon MSK uses the default security group associated with the VPC. If you specify security groups that were shared with you, you must ensure that you have permissions to them. Specifically, you need the ec2:DescribeSecurityGroups permission.

storageInfo

StorageInfo

False

Contains information about storage volumes attached to Amazon MSK broker nodes.

BrokerSoftwareInfo

Information about the current software installed on the cluster.

PropertyTypeRequiredDescription
configurationArn

string

False

The Amazon Resource Name (ARN) of the configuration used for the cluster. This field isn't visible in this preview release.

configurationRevision

integer

Format: int64

False

The revision of the configuration to use. This field isn't visible in this preview release.

kafkaVersion

string

False

The version of Apache Kafka. You can use Amazon MSK to create clusters that use Apache Kafka versions 1.1.1 and 2.2.1.

ClientAuthentication

Includes all client authentication information.

PropertyTypeRequiredDescription
sasl

Sasl

False

Details for client authentication using SASL. To turn on SASL, you must also turn on EncryptionInTransit by setting inCluster to true. You must set clientBroker to either TLS or TLS_PLAINTEXT. If you choose TLS_PLAINTEXT, then you must also set unauthenticated to true.

tls

Tls

False

Details for ClientAuthentication using TLS. To turn on TLS access control, you must also turn on EncryptionInTransit by setting inCluster to true and clientBroker to TLS.

unauthenticated

Unauthenticated

False

Details for ClientAuthentication using no authentication.

ClientBroker

Client-broker encryption in transit setting.

  • TLS

  • TLS_PLAINTEXT

  • PLAINTEXT

CloudWatchLogs

Details of the CloudWatch Logs destination for broker logs.

PropertyTypeRequiredDescription
enabled

boolean

True

Specifies whether broker logs get sent to the specified CloudWatch Logs destination.

logGroup

string

False

The CloudWatch log group that is the destination for broker logs.

ClusterInfo

Returns information about a cluster.

PropertyTypeRequiredDescription
activeOperationArn

string

False

Arn of active cluster operation.

brokerNodeGroupInfo

BrokerNodeGroupInfo

False

Information about the broker nodes.

clientAuthentication

ClientAuthentication

False

Includes all client authentication information.

clusterArn

string

False

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

clusterName

string

False

The name of the cluster.

creationTime

string

False

The time when the cluster was created.

currentBrokerSoftwareInfo

BrokerSoftwareInfo

False

Information about the version of software currently deployed on the brokers in the cluster.

currentVersion

string

False

The current version of the MSK cluster. Cluster versions aren't simple integers. You can obtain the current version by describing the cluster. An example version is KTVPDKIKX0DER.

encryptionInfo

EncryptionInfo

False

Includes all encryption-related information.

enhancedMonitoring

EnhancedMonitoring

False

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster. This property has three possible values: DEFAULT, PER_BROKER, and PER_TOPIC_PER_BROKER. For a list of the metrics associated with each of these three levels of monitoring, see Monitoring.

loggingInfo

LoggingInfo

False

You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.

numberOfBrokerNodes

integer

False

The number of broker nodes in the cluster.

openMonitoring

OpenMonitoring

False

Settings for open monitoring using Prometheus.

state

ClusterState

False

The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to MAINTENANCE when it starts the certificate-update operation. It sets it back to ACTIVE when the update is done. While a cluster is in the MAINTENANCE state, you can continue to produce and consume data, but you can't perform any update operations on it. You can perform update operations on a cluster when it is in the ACTIVE state.

stateInfo

StateInfo

False

Includes information of the cluster state.

storageMode

StorageMode

False

This controls storage mode for supported storage tiers.

tags

object

False

Tags attached to the cluster.

zookeeperConnectString

string

False

The connection string to use to connect to zookeeper cluster on plaintext port.

zookeeperConnectStringTls

string

False

The connection string to use to connect to the Apache ZooKeeper cluster on a TLS port.

ClusterState

The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to MAINTENANCE when it starts the certificate-update operation. It sets it back to ACTIVE when the update is done. While a cluster is in the MAINTENANCE state, you can continue to produce and consume data, but you can't perform any update operations on it. You can perform update operations on a cluster when it is in the ACTIVE state.

  • ACTIVE

  • CREATING

  • UPDATING

  • DELETING

  • FAILED

  • MAINTENANCE

  • REBOOTING_BROKER

  • HEALING

ConfigurationInfo

Specifies the configuration to use for the brokers.

PropertyTypeRequiredDescription
arn

string

True

ARN of the configuration to use.

revision

integer

Format: int64

Minimum: 1

True

The revision of the configuration to use.

ConnectivityInfo

Broker access controls.

PropertyTypeRequiredDescription
publicAccess

PublicAccess

False

Access control settings for the cluster's brokers.

CreateClusterRequest

Creates a cluster.

PropertyTypeRequiredDescription
brokerNodeGroupInfo

BrokerNodeGroupInfo

True

Information about the broker nodes in the cluster.

clientAuthentication

ClientAuthentication

False

Includes all client authentication related information.

clusterName

string

MinLength: 1

MaxLength: 64

True

The name of the cluster.

configurationInfo

ConfigurationInfo

False

Represents the configuration that you want MSK to use for the cluster.

encryptionInfo

EncryptionInfo

False

Includes all encryption-related information.

enhancedMonitoring

EnhancedMonitoring

False

Specifies the level of monitoring for the MSK cluster. The possible values are DEFAULT, PER_BROKER, and PER_TOPIC_PER_BROKER.

kafkaVersion

string

MinLength: 1

MaxLength: 128

True

The version of Apache Kafka. You can use Amazon MSK to create clusters that use Apache Kafka versions 1.1.1 and 2.2.1.

loggingInfo

LoggingInfo

False

Logging Info details.

numberOfBrokerNodes

integer

True

The number of broker nodes in the cluster.

openMonitoring

OpenMonitoringInfo

False

The settings for open monitoring.

storageMode

StorageMode

False

This controls storage mode for supported storage tiers.

tags

object

False

Create tags when creating the cluster.

CreateClusterResponse

Returns information about the created cluster.

PropertyTypeRequiredDescription
clusterArn

string

False

The Amazon Resource Name (ARN) of the cluster.

clusterName

string

False

The name of the MSK cluster.

state

ClusterState

False

The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to MAINTENANCE when it starts the certificate-update operation. It sets it back to ACTIVE when the update is done. While a cluster is in the MAINTENANCE state, you can continue to produce and consume data, but you can't perform any update operations on it. You can perform update operations on a cluster when it is in the ACTIVE state.

EBSStorageInfo

Contains information about the EBS storage volumes attached to the broker nodes.

PropertyTypeRequiredDescription
volumeSize

integer

Minimum: 1

Maximum: 16384

False

The size in GiB of the EBS volume for the data drive on each broker node.

EncryptionAtRest

The data-volume encryption details. You can't update encryption at rest settings for existing clusters.

PropertyTypeRequiredDescription
dataVolumeKMSKeyId

string

True

The ARN of the Amazon KMS key for encrypting data at rest. If you don't specify a KMS key, MSK creates one for you and uses it.

EncryptionInTransit

The settings for encrypting data in transit.

PropertyTypeRequiredDescription
clientBroker

ClientBroker

False

Indicates the encryption setting for data in transit between clients and brokers. You must set it to one of the following values.

TLS means that client-broker communication is enabled with TLS only.

TLS_PLAINTEXT means that client-broker communication is enabled for both TLS-encrypted, as well as plaintext data.

PLAINTEXT means that client-broker communication is enabled in plaintext only.

The default value is TLS.

inCluster

boolean

False

When set to true, it indicates that data communication among the broker nodes of the cluster is encrypted. When set to false, the communication happens in plaintext.

The default value is true.

EncryptionInfo

Includes encryption-related information, such as the Amazon KMS key used for encrypting data at rest and whether you want MSK to encrypt your data in transit.

PropertyTypeRequiredDescription
encryptionAtRest

EncryptionAtRest

False

The data-volume encryption details.

encryptionInTransit

EncryptionInTransit

False

The details for encryption in transit.

EnhancedMonitoring

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster. This property has three possible values: DEFAULT, PER_BROKER, and PER_TOPIC_PER_BROKER. For a list of the metrics associated with each of these three levels of monitoring, see Monitoring.

  • DEFAULT

  • PER_BROKER

  • PER_TOPIC_PER_BROKER

  • PER_TOPIC_PER_PARTITION

Error

Returns information about an error.

PropertyTypeRequiredDescription
invalidParameter

string

False

The parameter that caused the error.

message

string

False

The description of the error.

Firehose

Firehose details for BrokerLogs.

PropertyTypeRequiredDescription
deliveryStream

string

False

The Kinesis Data Firehose delivery stream that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get send to the specified Kinesis Data Firehose delivery stream.

IAM

Details for SASL/IAM client authentication.

PropertyTypeRequiredDescription
enabled

boolean

False

SASL/IAM authentication is enabled or not.

JmxExporter

Indicates whether you want to enable or disable the JMX Exporter.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

Indicates whether you want to enable or disable the JMX Exporter.

JmxExporterInfo

JMX Exporter details.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

JMX Exporter being enabled in broker.

ListClustersResponse

The response contains an array containing cluster information and a next token if the response is truncated.

PropertyTypeRequiredDescription
clusterInfoList

Array of type ClusterInfo

False

Information on each of the MSK clusters in the response.

nextToken

string

False

The paginated results marker. When the result of a ListClusters operation is truncated, the call returns NextToken in the response. To get another batch of clusters, provide this token in your next request.

LoggingInfo

You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.

PropertyTypeRequiredDescription
brokerLogs

BrokerLogs

True

You can configure your MSK cluster to send broker logs to different destination types. This configuration specifies the details of these destinations.

NodeExporter

Indicates whether you want to enable or disable the Node Exporter.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

Indicates whether you want to enable or disable the Node Exporter.

NodeExporterInfo

Node Exporter details.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

Node Exporter being enabled in broker.

OpenMonitoring

JMX and Node monitoring for the MSK cluster.

PropertyTypeRequiredDescription
prometheus

Prometheus

True

Prometheus exporter settings.

OpenMonitoringInfo

JMX and Node monitoring for cluster.

PropertyTypeRequiredDescription
prometheus

PrometheusInfo

True

Prometheus details.

Prometheus

Prometheus settings for open monitoring.

PropertyTypeRequiredDescription
jmxExporter

JmxExporter

False

Indicates whether you want to enable or disable the JMX Exporter.

nodeExporter

NodeExporter

False

Indicates whether you want to enable or disable the Node Exporter.

PrometheusInfo

Prometheus details.

PropertyTypeRequiredDescription
jmxExporter

JmxExporterInfo

False

JMX Exporter details.

nodeExporter

NodeExporterInfo

False

Node Exporter details.

PublicAccess

Broker access controls

PropertyTypeRequiredDescription
type

string

False

DISABLED means that public access is turned off. SERVICE_PROVIDED_EIPS means that public access is turned on.

S3

The details of the Amazon S3 destination for broker logs.

PropertyTypeRequiredDescription
bucket

string

False

The name of the S3 bucket that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get sent to the specified Amazon S3 destination.

prefix

string

False

The S3 prefix that is the destination for broker logs.

Sasl

Details for client authentication using SASL. To turn on SASL, you must also turn on EncryptionInTransit by setting inCluster to true. You must set clientBroker to either TLS or TLS_PLAINTEXT. If you choose TLS_PLAINTEXT, then you must also set unauthenticated to true.

PropertyTypeRequiredDescription
iam

IAM

False

Details for ClientAuthentication using IAM.

scram

Scram

False

Details for SASL/SCRAM client authentication.

Scram

Details for SASL/SCRAM client authentication.

PropertyTypeRequiredDescription
enabled

boolean

False

SASL/SCRAM authentication is enabled or not.

StateInfo

Includes information about the state of the cluster.

PropertyTypeRequiredDescription
code

string

False

If the cluster is in an unusable state, this field contains the code that describes the issue.

message

string

False

If the cluster is in an unusable state, this field contains a message that describes the issue.

StorageInfo

Contains information about storage volumes attached to Amazon MSK broker nodes.

PropertyTypeRequiredDescription
ebsStorageInfo

EBSStorageInfo

False

EBS volume information.

StorageMode

Controls storage mode for various supported storage tiers.

  • LOCAL

  • TIERED

Tls

Details for client authentication using TLS.

PropertyTypeRequiredDescription
certificateAuthorityArnList

Array of type string

False

List of Amazon Private CA ARNs.

enabled

boolean

False

TLS authentication is enabled or not.

Unauthenticated

Details for allowing no client authentication.

PropertyTypeRequiredDescription
enabled

boolean

False

Unauthenticated is enabled or not.