Updating the broker size - Amazon Managed Streaming for Apache Kafka
Updating the broker size using the Amazon Web Services Management ConsoleUpdating the broker size using the Amazon CLIUpdating the broker size using the API
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Updating the broker size

You can scale your MSK cluster on demand by changing the size of your brokers without reassigning Apache Kafka partitions. Changing the size of your brokers gives you the flexibility to adjust your MSK cluster's compute capacity based on changes in your workloads, without interrupting your cluster I/O. Amazon MSK uses the same broker size for all the brokers in a given cluster.

This section describes how to update the broker size for your MSK cluster. You can update your cluster broker size from M5 or T3 to M7g, or from M7g to M5. Be aware that migrating to a smaller broker size can decrease performance and reduce maxiumum achievable throughput per broker. Migrating to a larger broker size can increase performance but may cost more.

The broker-size update happens in a rolling fashion while the cluster is up and running. This means that Amazon MSK takes down one broker at a time to perform the broker-size update. For information about how to make a cluster highly available during a broker-size update, see Build highly available clusters. To further reduce any potential impact on productivity, you can perform the broker-size update during a period of low traffic.

During a broker-size update, you can continue to produce and consume data. However, you must wait until the update is done before you can reboot brokers or invoke any of the update operations listed under Amazon MSK operations.

If you want to update your cluster to a smaller broker size, we recommend that you try the update on a test cluster first to see how it affects your scenario.

Important

You can't update a cluster to a smaller broker size if the number of partitions per broker exceeds the maximum number specified in Right-size your cluster: Number of partitions per broker.

Updating the broker size using the Amazon Web Services Management Console

  1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/.

  2. Choose the MSK cluster for which you want to update the broker size.

  3. On the details page for the cluster, find the Brokers summary section, and choose Edit broker size.

  4. Choose the broker size you want from the list.

  5. Save changes.

Updating the broker size using the Amazon CLI

  1. Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can find it by listing all clusters. For more information, see Listing Amazon MSK clusters.

    Replace Current-Cluster-Version with the current version of the cluster and TargetType with the new size that you want the brokers to be. To learn more about broker sizes, see Broker sizes.

    aws kafka update-broker-type --cluster-arn ClusterArn --current-version Current-Cluster-Version --target-instance-type TargetType

    The following is an example of how to use this command:

    aws kafka update-broker-type --cluster-arn "arn:aws:kafka:us-east-1:0123456789012:cluster/exampleName/abcd1234-0123-abcd-5678-1234abcd-1" --current-version "K1X5R6FKA87" --target-instance-type kafka.m5.large

    The output of this command looks like the following JSON example.

    {
    "ClusterArn": "arn:aws:kafka:us-east-1:0123456789012:cluster/exampleName/abcd1234-0123-abcd-5678-1234abcd-1",
    "ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef"
}

  2. To get the result of the update-broker-type operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update-broker-type command.

    aws kafka describe-cluster-operation --cluster-operation-arn ClusterOperationArn

    The output of this describe-cluster-operation command looks like the following JSON example.

    {
  "ClusterOperationInfo": {
    "ClientRequestId": "982168a3-939f-11e9-8a62-538df00285db",
    "ClusterArn": "arn:aws:kafka:us-east-1:0123456789012:cluster/exampleName/abcd1234-0123-abcd-5678-1234abcd-1",
    "CreationTime": "2021-01-09T02:24:22.198000+00:00",
    "OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd-abcd-4f7f-1234-9876543210ef",
    "OperationState": "UPDATE_COMPLETE",
    "OperationType": "UPDATE_BROKER_TYPE",
    "SourceClusterInfo": {
      "InstanceType": "t3.small"
    },
    "TargetClusterInfo": {
      "InstanceType": "m5.large"
    }
  }
}

    If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe-cluster-operation command again.

Updating the broker size using the API

To update the broker size using the API, see UpdateBrokerType.

You can use UpdateBrokerType to update your cluster broker size from M5 or T3 to M7g, or from M7g to M5.