View a markdown version of this page

Troubleshoot issues in Amazon MSK Connect - Amazon Managed Streaming for Apache Kafka
Kafka Connect error codes
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshoot issues in Amazon MSK Connect

The following information can help you troubleshoot problems that you might have while using MSK Connect. You can also post your issue to the Amazon Web Services re:Post.

Connector is unable to access resources hosted on the public internet

See Enabling internet access for Amazon MSK Connect.

Connector's number of running tasks is not equal to the number of tasks specified in tasks.max

Here are some reasons a connector may use fewer tasks than the specified tasks.max configuration:

  • Some connector implementations limit the number of tasks the can be used. For example, the Debezium connector for MySQL is limited to using a single task.

  • When using autoscaled capacity mode, Amazon MSK Connect overrides a connector's tasks.max property with a value that is proportional to the number of workers running in the connector and the number of MCUs per worker. If you have configured the optional maxAutoscalingTaskCount parameter, the tasks.max value will not exceed this limit. For more information, see Understand maximum autoscaling task count.

  • For sink connectors, the level of parallelism (number of tasks) cannot be more than the number of topic partitions. While you can set the tasks.max larger than that, a single partition is never processed by more than a single task at a time.

  • In Kafka Connect 2.7.x, the default consumer partition assignor is RangeAssignor. The behavior of this assignor is to give the first partition of every topic to a single consumer, the second partition of every topic to a single consumer, etc. This means that the maximum number of active tasks for a sink connector using RangeAssignor is equal to the maximum number of partitions in any single topic being consumed. If this doesn't work for your use case, you should create a Worker Configuration in which the consumer.partition.assignment.strategy property is set to a more suitable consumer partition assignor. See Kafka 2.7 Interface ConsumerPartitionAssignor: All Known Implementing Classes.

Connector restart operation reaches RESTART_FAILED state

If a restart operation reaches the RESTART_FAILED state, try the following to resolve the issue:

  • Check the errorInfo field in the DescribeConnectorOperation response for details.

  • Review connector logs from CloudWatch, Amazon S3, or Firehose for error messages indicating why tasks cannot complete startup.

  • Check whether the connector's tasks are blocked by external dependencies (databases, network endpoints, or other systems that the connector interacts with).

  • Verify that the connector's Kafka cluster is reachable and healthy.

Cannot restart connector — 400 BadRequestException

Here are some reasons why you might not be able to restart a connector:

  • The connector is not in the RUNNING state (for example, it is CREATING, UPDATING, or DELETING).

  • Another lifecycle operation is already in progress on the connector.

  • You specified --only-failed-tasks on a connector using Amazon MSK version earlier than 3.7 (this feature requires version 3.7 or later).

  • The connector was created before the restart capability was available. This feature is currently supported only for newly created connectors.

Connector pause or resume operation reaches a failed state

A pause operation that reaches the PAUSE_FAILED state, or a resume operation that reaches the RESUME_FAILED state, does not always leave your connector unusable, and only some failures can be retried. Diagnose the failure as follows:

  1. Call DescribeConnectorOperation with the connectorOperationArn that your pause or resume request returned, and read the errorInfo field to find out why the operation failed.

  2. Call DescribeConnector and note the connector's current state. The state determines what you can do next.

  3. Review your connector logs in CloudWatch, Amazon S3, or Firehose for the period when the operation ran. Look for errors that stop the connector from applying the change, such as an unreachable Kafka cluster, a failed broker authentication, or a task that is blocked on an external dependency such as a source database or a sink target.

Then take the action for the state that your connector is in.

RUNNING after a failed pause, or PAUSED after a failed resume

The change never reached your connector, so the connector is still in the state that it started in, and the operation is safe to retry. Fix the cause that errorInfo and your connector logs point to. For example, update the connector to correct a configuration value, or restore network access to your Kafka cluster. Then call PauseConnector or ResumeConnector again.

PAUSED after a failed pause, or RUNNING after a failed resume

Your connector reached the state that you asked for, but some of its tasks did not, and errorInfo reports KafkaConnect.ConnectorNotReachedTargetState. Do not retry the operation. The connector is already in the target state, so the request is rejected with a 400 BadRequestException. Instead, compare the PausedTaskCount, RunningTaskCount, and ErroredTaskCount metrics against the number of tasks that you expect, and use your connector logs to find why the remaining tasks did not converge.

To bring the remaining tasks into the state that you asked for, first fix the cause that your connector logs report. Then update the connector with skipHealthValidation set to true. The update restarts the connector's tasks, and they then apply the pause or resume.

After a failed resume, you can also restart the connector to restart the tasks that did not start, because the connector is in the RUNNING state that a restart requires.

PAUSING or RESUMING

First confirm that the operation has actually failed: call DescribeConnectorOperation and check the operation state, because a connector stays in PAUSING or RESUMING while the operation is still in progress too. If the operation state is PAUSE_FAILED or RESUME_FAILED and the connector is still PAUSING or RESUMING, Amazon MSK Connect could not confirm whether your connector accepted the change. You cannot retry the operation, and you cannot run any other lifecycle operation, because the connector is not in a state that any of them accept. Contact Amazon Support to recover the connector.

For what each Kafka Connect error code means, see Kafka Connect error codes.

For more information about where a failed operation leaves your connector, see If a pause or resume operation fails.

Connector pause or resume operation stays in progress

An operation can stay in the PAUSE_IN_PROGRESS or RESUME_IN_PROGRESS state for a long time. If Amazon MSK Connect cannot reach your connector's Kafka Connect API, or cannot confirm that your connector applied the change, it retries the operation, so an operation that is still in progress can still reach PAUSE_COMPLETE or RESUME_COMPLETE on a later attempt. An operation that has not reached a terminal state has not failed.

A connector is left in the PAUSING or RESUMING state both by an operation that is still in progress and by one that failed, so call DescribeConnectorOperation to tell the two apart before you act. While the operation state is PAUSE_IN_PROGRESS or RESUME_IN_PROGRESS, the operation is still running, and:

  • Your connector stays in the PAUSING or RESUMING state.

  • You cannot perform other lifecycle operations on your connector, such as update or delete.

  • You cannot retry the pause or resume request, because your connector is not in the state that the operation requires.

Because you cannot retry, remove whatever is stopping your connector from applying the change so that the next retry succeeds. Check your connector logs in CloudWatch, Amazon S3, or Firehose for an unreachable Kafka cluster, a failed broker authentication, or a task that is blocked on an external dependency, and fix what you find. Keep polling DescribeConnectorOperation. If the operation does not reach a terminal state, contact Amazon Support to recover the connector.

Cannot pause or resume a connector: 400 BadRequestException

Here are some reasons why you might not be able to pause or resume a connector:

  • You called PauseConnector on a connector that is not in the RUNNING state. A connector that is already PAUSED or PAUSING cannot be paused again. Call DescribeConnector to confirm the connector's state, and pause it only from RUNNING.

  • You called ResumeConnector on a connector that is not in the PAUSED state. A connector that is already RESUMING is being resumed, so wait for that operation to complete instead of calling ResumeConnector again.

  • Another lifecycle operation is already in progress on your connector. Track that operation with DescribeConnectorOperation, wait for it to reach a terminal state, and then submit your request.

Cannot update a paused connector

To update a connector that is in the PAUSED state, set skipHealthValidation to true in the update request, because Amazon MSK Connect cannot validate the health of a connector that is not processing records.

Worker settings update reaches UPDATE_FAILED state

A worker settings update that reaches the UPDATE_FAILED state did not take effect. The connector keeps its previous capacity, plugin, and worker configuration, and Amazon MSK Connect either did not start the update or rolled it back. Call DescribeConnectorOperation with the connectorOperationArn that your update request returned, and read the errorInfo field. Here are some common causes:

  • The connector or its tasks did not become healthy on the new workers. For example, the new plugin does not contain the connector class that your connector configuration specifies, or the new worker configuration is not compatible with the connector. Review your connector logs in CloudWatch, Amazon S3, or Firehose for the period when the update ran. Fix the plugin or worker configuration, and then submit the update again. An update that skips health validation stays in effect even when the connector is unhealthy on the new workers. For more information about health validation, see Health validation and rollback.

  • errorInfo reports KafkaConnect.NotFound, KafkaConnect.BrokerUnreachable, or KafkaConnect.BrokerAuthenticationFailure. Amazon MSK Connect checks that the connector is registered on its workers and that it can reach your Kafka cluster before it starts a plugin or worker configuration update, so it did not start this update. For what each code means and how to resolve it, see Kafka Connect error codes. Then retry the update.

  • errorInfo reports KafkaConnect.BrokerNamesUnresolvable. The connector cannot resolve the names of your Kafka cluster's brokers, so Amazon MSK Connect did not start the update. Verify that the bootstrap servers in your connector's configuration are correct, and that the connector's subnets can resolve them. Then retry the update.

  • errorInfo reports InvalidInput.InsufficientFreeAddressesInSubnet. Your subnets do not have enough free IP addresses for the new workers. During a plugin or worker configuration update, the previous workers and the new workers exist at the same time, and each worker uses one IP address. Free up IP addresses in the connector's subnets, and then retry the update.

Worker settings update stays in UPDATE_IN_PROGRESS state

An update takes time while Amazon MSK Connect applies the new worker settings. For a plugin or worker configuration change, that includes starting the new workers and moving the connector's tasks to them. While the operation is in the UPDATE_IN_PROGRESS state, the connector is UPDATING, and you cannot retry the update or run other lifecycle operations on the connector. If the operation does not reach the UPDATE_COMPLETE or UPDATE_FAILED state and the connector stays UPDATING, contact Amazon Support to recover the connector.

Cannot update worker settings: 400 BadRequestException

Here are some reasons why you might not be able to update a connector's worker settings:

  • The connector is not in the RUNNING or PAUSED state (for example, another lifecycle operation is in progress and the connector is UPDATING).

  • The request specifies connectorConfiguration together with a worker setting field (capacity, plugins, workerConfiguration, or useDefaultWorkerConfiguration). These two kinds of update require separate requests.

  • The request specifies both workerConfiguration and useDefaultWorkerConfiguration.

  • The request specifies more than one plugin. A connector uses exactly one plugin.

  • The values in the request are the same as the connector's current worker settings.

  • The currentVersion in the request is not the connector's current version. DescribeConnector returns the current version.

  • The custom plugin is not in the ACTIVE state, belongs to a different account or Region, or the revision that you specified does not exist.

  • The worker configuration belongs to a different account or Region, or the revision that you specified does not exist.

Kafka Connect error codes

When a connector lifecycle operation fails, the errorInfo field in the DescribeConnectorOperation response can report one of the following Kafka Connect error codes.

KafkaConnect.ConnectorNotReachedTargetState

Your connector accepted the change, but some of its tasks did not reach the target state in time. The connector is in the state that you asked for, so work through the task counts and the connector logs rather than retrying the operation.

KafkaConnect.NotFound

Your connector is no longer registered on its Kafka Connect workers. This can happen for several reasons, for example if the Kafka Connect internal topics that store connector configurations are deleted. To recreate the connector, update its configuration. Updating the connector does not always bring it back in the state that you wanted, so check the connector's state and its task-count metrics before you run another operation on it.

KafkaConnect.BrokerAuthenticationFailure

Amazon MSK Connect could not authenticate to your Kafka cluster. Verify the authentication type that your connector uses, and verify that the permissions that this type requires are still in place. Then retry the operation.

KafkaConnect.BrokerUnreachable

Amazon MSK Connect could not reach your Kafka cluster. Verify that the cluster is healthy, that its bootstrap servers match your connector's configuration, and that the subnets and security groups that your connector uses still allow traffic to the cluster. Then retry the operation.