Requirements and considerations for creating an Amazon MSK Replicator - Amazon Managed Streaming for Apache Kafka
Permissions required to create an MSK ReplicatorSupported cluster types and versionsMSK Serverless cluster configurationCluster configuration changes
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Requirements and considerations for creating an Amazon MSK Replicator

Note these MSK cluster requirements for running an Amazon MSK Replicator.

Permissions required to create an MSK Replicator

Here is an example of the IAM policy required to create an MSK Replicator. The action kafka:TagResource is only needed if tags are provided when creating the MSK Replicator. Replicator IAM policies should be attached to the IAM role that corresponds to your client. For information on creating authorization policies, see Create authorization policies.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "iam:CreateServiceLinkedRole",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:CreateNetworkInterface",
                "ec2:DescribeVpcs",
                "kafka:CreateReplicator",
                "kafka:TagResource"
            ],
            "Resource": "*"
        }
    ]
}

The following is an example IAM policy to describe replicator. Either the kafka:DescribeReplicator action or kafka:ListTagsForResource action is needed, not both.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "kafka:DescribeReplicator",
                "kafka:ListTagsForResource"
            ],
            "Resource": "*"
        }
    ]
}

Supported cluster types and versions

These are requirements for supported instance types, Kafka versions, and network configurations.

  • MSK Replicator supports both MSK provisioned clusters and MSK Serverless clusters in any combination as source and target clusters. Other types of Kafka clusters are not supported at this time by MSK Replicator.

  • MSK Serverless clusters require IAM access control, don't support Apache Kafka ACL replication and with limited support on-topic configuration replication. See MSK Serverless.

  • MSK Replicator is supported only on clusters running Apache Kafka 2.7.0 or higher, regardless of whether your source and target clusters are in the same or in different Amazon Regions.

  • MSK Replicator supports clusters using instance types of m5.large or larger. t3.small clusters are not supported.

  • If you are using MSK Replicator with an MSK Provisioned cluster, you need a minimum of three brokers in both source and target clusters. You can replicate data across clusters in two Availability Zones, but you would need a minimum of four brokers in those clusters.

  • Both your source and target MSK clusters must be in the same Amazon account. Replication across clusters in different accounts is not supported.

  • If the source and target MSK clusters are in different Amazon Regions (cross-region), MSK Replicator requires the source cluster to have multi-VPC private connectivity turned on for its IAM Access Control method. Multi-VPC is not required for other authentication methods on the source cluster. Multi-VPC is not required if you are replicating data between clusters in the same Amazon Region. See Amazon MSK multi-VPC private connectivity in a single Region.

  • Identical topic name replication (Keep the same topics name in console) requires an MSK cluster running Kafka version 2.8.1 or higher.

  • For Identical topic name replication (Keep the same topics name in console) configurations, to avoid the risk of cyclic replication, do not make changes to the headers that MSK Replicator creates (__mskmr).

MSK Serverless cluster configuration

  • MSK Serverless supports replication of these topic configurations for MSK Serverless target clusters during topic creation: cleanup.policy, compression.type, max.message.bytes, retention.bytes, retention.ms.

  • MSK Serverless supports only these topic configurations during topic configuration sync: compression.type, max.message.bytes, retention.bytes, retention.ms.

  • Replicator uses 83 compacted partitions on target MSK Serverless clusters. Make sure that target MSK Serverless clusters have a sufficient number of compacted partitions. See MSK Serverless quota.

Cluster configuration changes

  • It’s recommended that you do not turn tiered storage on or off after the MSK Replicator has been created. If your target cluster is not tiered, then MSK won’t copy the tiered storage configurations, regardless of whether your source cluster is tiered or not. If you turn on tiered storage on the target cluster after Replicator is created, the Replicator needs to be recreated. If you want to copy data from a non-tiered to a tiered cluster, you should not copy topic configurations. See Enabling and disabling tiered storage on an existing topic.

  • Don’t change cluster configuration settings after MSK Replicator creation. Cluster configuration settings are validated during MSK Replicator creation. To avoid problems with the MSK Replicator, don’t change the following settings after the MSK Replicator is created.

    • Change MSK cluster to t3 instance type.

    • Change service execution role permissions.

    • Disable MSK multi-VPC private connectivity.

    • Change the attached cluster resource-based policy.

    • Change cluster security group rules.