Amazon MSK configuration operations - Amazon Managed Streaming for Apache Kafka
Create configurationTo update an MSK configurationTo delete an MSK configurationTo describe an MSK configurationTo describe an MSK configuration revisionTo list all MSK configurations in your account for the current Region
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon MSK configuration operations

This topic describes how to create custom MSK configurations and how to perform operations on them. For information about how to use MSK configurations to create or update clusters, see Amazon MSK: How it works.

To create an MSK configuration

  1. Create a file where you specify the configuration properties that you want to set and the values that you want to assign to them. The following are the contents of an example configuration file.

    auto.create.topics.enable = true

log.roll.ms = 604800000

  2. Run the following Amazon CLI command, and replace config-file-path with the path to the file where you saved your configuration in the previous step.

    Note

    The name that you choose for your configuration must match the following regex: "^[0-9A-Za-z][0-9A-Za-z-]{0,}$".

    aws kafka create-configuration --name "ExampleConfigurationName" --description "Example configuration description." --kafka-versions "1.1.1" --server-properties fileb://config-file-path

    The following is an example of a successful response after you run this command.

    {
    "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-1234-abcd-1234-abcd123e8e8e-1",
    "CreationTime": "2019-05-21T19:37:40.626Z",
    "LatestRevision": {
        "CreationTime": "2019-05-21T19:37:40.626Z",
        "Description": "Example configuration description.",
        "Revision": 1
    },
    "Name": "ExampleConfigurationName"
}

  3. The previous command returns an Amazon Resource Name (ARN) for your new configuration. Save this ARN because you need it to refer to this configuration in other commands. If you lose your configuration ARN, you can list all the configurations in your account to find it again.

To update an MSK configuration

  1. Create a file where you specify the configuration properties that you want to update and the values that you want to assign to them. The following are the contents of an example configuration file.

    auto.create.topics.enable = true

min.insync.replicas = 2

  2. Run the following Amazon CLI command, and replace config-file-path with the path to the file where you saved your configuration in the previous step.

    Replace configuration-arn with the ARN that you obtained when you created the configuration. If you didn't save the ARN when you created the configuration, you can use the list-configurations command to list all configuration in your account. The configuration that you want in the list appears in the response. The ARN of the configuration also appears in that list.

    aws kafka update-configuration --arn configuration-arn --description "Example configuration revision description." --server-properties fileb://config-file-path

  3. The following is an example of a successful response after you run this command.

    {
    "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-1234-abcd-1234-abcd123e8e8e-1",
    "LatestRevision": {
        "CreationTime": "2020-08-27T19:37:40.626Z",
        "Description": "Example configuration revision description.",
        "Revision": 2
    }
}

To delete an MSK configuration

The following procedure shows how to delete a configuration that isn't attached to a cluster. You can't delete a configuration that's attached to a cluster.

  1. To run this example, replace configuration-arn with the ARN that you obtained when you created the configuration. If you didn't save the ARN when you created the configuration, you can use the list-configurations command to list all configuration in your account. The configuration that you want in the list appears in the response. The ARN of the configuration also appears in that list.

    aws kafka delete-configuration --arn configuration-arn

  2. The following is an example of a successful response after you run this command.

    {
    "arn": " arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-1234-abcd-1234-abcd123e8e8e-1",
    "state": "DELETING"
}

To describe an MSK configuration

  1. The following command returns metadata about the configuration. To get a detailed description of the configuration, run the describe-configuration-revision.

    To run this example, replace configuration-arn with the ARN that you obtained when you created the configuration. If you didn't save the ARN when you created the configuration, you can use the list-configurations command to list all configuration in your account. The configuration that you want in the list appears in the response. The ARN of the configuration also appears in that list.

    aws kafka describe-configuration --arn configuration-arn

  2. The following is an example of a successful response after you run this command.

    {
    "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1",
    "CreationTime": "2019-05-21T00:54:23.591Z",
    "Description": "Example configuration description.",
    "KafkaVersions": [
        "1.1.1"
    ],
    "LatestRevision": {
        "CreationTime": "2019-05-21T00:54:23.591Z",
        "Description": "Example configuration description.",
        "Revision": 1
    },
    "Name": "SomeTest"
}

To describe an MSK configuration revision

If you use the describe-configuration command to describe an MSK configuration, you see the metadata of the configuration. To get a description of the configuration, use the command, describe-configuration-revision.

  • Run the following command and replace configuration-arn with the ARN that you obtained when you created the configuration. If you didn't save the ARN when you created the configuration, you can use the list-configurations command to list all configuration in your account. The configuration that you want in the list that appears in the response. The ARN of the configuration also appears in that list.

    aws kafka describe-configuration-revision --arn configuration-arn --revision 1

    The following is an example of a successful response after you run this command.

    {
    "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1",
    "CreationTime": "2019-05-21T00:54:23.591Z",
    "Description": "Example configuration description.",
    "Revision": 1,
    "ServerProperties": "YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZSA9IHRydWUKCgp6b29rZWVwZXIuY29ubmVjdGlvbi50aW1lb3V0Lm1zID0gMTAwMAoKCmxvZy5yb2xsLm1zID0gNjA0ODAwMDAw"
}

    The value of ServerProperties is encoded with base64. If you use a base64 decoder (for example, https://www.base64decode.org/) to decode it manually, you get the contents of the original configuration file that you used to create the custom configuration. In this case, you get the following:

    auto.create.topics.enable = true

log.roll.ms = 604800000

To list all MSK configurations in your account for the current Region

  • Run the following command.

    aws kafka list-configurations

    The following is an example of a successful response after you run this command.

    {
    "Configurations": [
        {
            "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1",
            "CreationTime": "2019-05-21T00:54:23.591Z",
            "Description": "Example configuration description.",
            "KafkaVersions": [
                "1.1.1"
            ],
            "LatestRevision": {
                "CreationTime": "2019-05-21T00:54:23.591Z",
                "Description": "Example configuration description.",
                "Revision": 1
            },
            "Name": "SomeTest"
        },
        {
            "Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/SomeTest/abcdabcd-1234-abcd-1234-abcd123e8e8e-1",
            "CreationTime": "2019-05-03T23:08:29.446Z",
            "Description": "Example configuration description.",
            "KafkaVersions": [
                "1.1.1"
            ],
            "LatestRevision": {
                "CreationTime": "2019-05-03T23:08:29.446Z",
                "Description": "Example configuration description.",
                "Revision": 1
            },
            "Name": "ExampleConfigurationName"
        }
    ]
}