Monitoring Properties - Amazon Managed Streaming for Apache Kafka

Monitoring Properties

The options for monitoring an Amazon MSK cluster. You can specify which Apache Kafka metrics you want Amazon MSK to gather and send to Amazon CloudWatch. You can also configure open monitoring to gather metrics with Prometheus or Prometheus-compatible tools.

URI

/v1/clusters/clusterArn/monitoring

HTTP methods

PUT

Operation ID: UpdateMonitoring

Updates the monitoring settings for the cluster. You can use this operation to specify which Apache Kafka metrics you want Amazon MSK to send to Amazon CloudWatch. You can also specify settings for open monitoring with Prometheus. The following Python 3.6 example enables open monitoring with the Node Exporter. It also sets enhanced monitoring to PER_BROKER. For more information about monitoring, see Monitoring.

import boto3 import time client = boto3.client('kafka') update_monitoring_response = client.update_monitoring( ClusterArn='arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4', CurrentVersion='K12V3IB1VIZHHY', EnhancedMonitoring='PER_BROKER', OpenMonitoring={"Prometheus":{"JmxExporter":{"EnabledInBroker":False},"NodeExporter":{"EnabledInBroker":True}}} ) operation_arn = update_monitoring_response['ClusterOperationArn'] print('The ARN of the update operation is ' + operation_arn) describe_cluster_operation_response = client.describe_cluster_operation(ClusterOperationArn=operation_arn) operation_state = describe_cluster_operation_response['ClusterOperationInfo']['OperationState'] print('The status of the update operation is ' + operation_state) updated = False while not updated: print('Sleeping for 15 seconds before checking to see if the monitoring update is done...') time.sleep(15) describe_cluster_operation_response = client.describe_cluster_operation(ClusterOperationArn=operation_arn) operation_state = describe_cluster_operation_response['ClusterOperationInfo']['OperationState'] if 'UPDATE_COMPLETE' == operation_state: updated = True print('You have successfully updated the monitoring settings.')
Path parameters
NameTypeRequiredDescription
clusterArnStringTrue

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

Responses
Status codeResponse modelDescription
200 UpdateMonitoringRequest

Successful response.

400Error

The request isn't valid because the input is incorrect. Correct your input and then submit it again.

401Error

The request is not authorized. The provided credentials couldn't be validated.

403Error

Access forbidden. Check your credentials and then retry your request.

404Error

The resource could not be found due to incorrect input. Correct the input, then retry the request.

429Error

429 response

500Error

There was an unexpected internal server error. Retrying your request might resolve the issue.

503Error

503 response

OPTIONS

Enable CORS by returning the correct headers.

Path parameters
NameTypeRequiredDescription
clusterArnStringTrue

The Amazon Resource Name (ARN) that uniquely identifies the cluster.

Responses
Status codeResponse modelDescription
200None

Default response for CORS method

Schemas

Request bodies

{ "enhancedMonitoring": enum, "loggingInfo": { "brokerLogs": { "s3": { "bucket": "string", "prefix": "string", "enabled": boolean }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "cloudWatchLogs": { "logGroup": "string", "enabled": boolean } } }, "openMonitoring": { "prometheus": { "nodeExporter": { "enabledInBroker": boolean }, "jmxExporter": { "enabledInBroker": boolean } } }, "currentVersion": "string" }

Response bodies

{ "clusterArn": "string", "clusterOperationArn": "string" }
{ "message": "string", "invalidParameter": "string" }

Properties

BrokerLogs

The broker logs configuration for this MSK cluster.

PropertyTypeRequiredDescription
cloudWatchLogs

CloudWatchLogs

False

Details of the CloudWatch Logs destination for broker logs.

firehose

Firehose

False

Details of the Kinesis Data Firehose delivery stream that is the destination for broker logs.

s3

S3

False

Details of the Amazon S3 destination for broker logs.

CloudWatchLogs

Details of the CloudWatch Logs destination for broker logs.

PropertyTypeRequiredDescription
enabled

boolean

True

Specifies whether broker logs get sent to the specified CloudWatch Logs destination.

logGroup

string

False

The CloudWatch log group that is the destination for broker logs.

EnhancedMonitoring

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster. This property has three possible values: DEFAULT, PER_BROKER, and PER_TOPIC_PER_BROKER. For a list of the metrics associated with each of these three levels of monitoring, see Monitoring.

  • DEFAULT

  • PER_BROKER

  • PER_TOPIC_PER_BROKER

  • PER_TOPIC_PER_PARTITION

Error

Returns information about an error.

PropertyTypeRequiredDescription
invalidParameter

string

False

The parameter that caused the error.

message

string

False

The description of the error.

Firehose

Firehose details for BrokerLogs.

PropertyTypeRequiredDescription
deliveryStream

string

False

The Kinesis Data Firehose delivery stream that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get sent to the specified Kinesis Data Firehose delivery stream.

JmxExporterInfo

JMX Exporter details.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

JMX Exporter being enabled in broker.

LoggingInfo

You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.

PropertyTypeRequiredDescription
brokerLogs

BrokerLogs

True

You can configure your MSK cluster to send broker logs to different destination types. This configuration specifies the details of these destinations.

NodeExporterInfo

Node Exporter details.

PropertyTypeRequiredDescription
enabledInBroker

boolean

True

Node Exporter being enabled in broker.

OpenMonitoringInfo

JMX and Node monitoring for cluster.

PropertyTypeRequiredDescription
prometheus

PrometheusInfo

True

Prometheus details.

PrometheusInfo

Prometheus details.

PropertyTypeRequiredDescription
jmxExporter

JmxExporterInfo

False

JMX Exporter details.

nodeExporter

NodeExporterInfo

False

Node Exporter details.

S3

The details of the Amazon S3 destination for broker logs.

PropertyTypeRequiredDescription
bucket

string

False

The name of the S3 bucket that is the destination for broker logs.

enabled

boolean

True

Specifies whether broker logs get sent to the specified Amazon S3 destination.

prefix

string

False

The S3 prefix that is the destination for broker logs.

UpdateMonitoringRequest

Request body for UpdateMonitoring.

PropertyTypeRequiredDescription
currentVersion

string

True

The version of the MSK cluster to update. Cluster versions aren't simple numbers. You can describe an MSK cluster to find its version. When this update operation is successful, it generates a new cluster version.

enhancedMonitoring

EnhancedMonitoring

False

Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster.

loggingInfo

LoggingInfo

False

LoggingInfo details.

openMonitoring

OpenMonitoringInfo

False

The settings for open monitoring.

UpdateMonitoringResponse

Request body for UpdateMonitoring.

PropertyTypeRequiredDescription
clusterArn

string

False

The Amazon Resource Name (ARN) of the cluster.

clusterOperationArn

string

False

The Amazon Resource Name (ARN) of the cluster operation.

See also

For more information about using this API in one of the language-specific AWS SDKs and references, see the following:

UpdateMonitoring