Amazon Kinesis Data Firehose
开发人员指南
AWS 文档中描述的 AWS 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅中国的 AWS 服务入门

使用 CloudWatch Logs 监控 Kinesis Data Firehose

Kinesis Data Firehose 与 Amazon CloudWatch Logs 集成,以便您能在数据转换或数据传输的 Lambda 调用失败时查看特定错误日志。当您创建传输流时,可以启用 Kinesis Data Firehose 错误日志记录。

在 Kinesis Data Firehose 控制台中启用 Kinesis Data Firehose 错误日志记录时,系统会代表您为传输流创建日志组和相应的日志流。日志组名称的格式为 /aws/kinesisfirehose/delivery-stream-name,其中,delivery-stream-name 是相应的传输流的名称。日志流名称为 S3DeliveryRedshiftDeliveryElasticsearchDelivery,具体视传输目标而定。数据转换的 Lambda 调用错误还会记录到用于数据传输错误的日志流。

例如,如果您创建将 Amazon Redshift 作为目标的传输流“MyStream”并启用 Kinesis Data Firehose 错误日志记录,则系统会代表您创建以下各项:一个名为 aws/kinesisfirehose/MyStream 的日志组和两个名为 S3DeliveryRedshiftDelivery 的日志流。在该示例中,日志流 S3Delivery 用于记录与传输到中间 S3 存储桶失败有关的错误。日志流 RedshiftDelivery 用于记录与 Lambda 调用失败以及传输到 Amazon Redshift 集群失败有关的错误。

如果您通过 AWS CLI 或 AWS SDK(使用 CloudWatchLoggingOptions 配置)启用 Kinesis Data Firehose 错误日志记录,必须提前创建日志组和日志流。我们建议您将该日志组和日志流专门用于记录 Kinesis Data Firehose 错误。此外,还应确保关联的 IAM 策略拥有 "logs:putLogEvents" 权限。有关更多信息,请参阅使用 Amazon Kinesis Data Firehose 控制访问

注意,Kinesis Data Firehose 不保证将所有传输错误日志都发送到 CloudWatch Logs。在传输故障率很高的情况下,Kinesis Data Firehose 先对传输错误日志进行采样,然后将它们发送到 CloudWatch Logs。

对于发送到 CloudWatch Logs 的错误日志,存在象征性的少许费用。有关更多信息,请参阅Amazon CloudWatch 定价

数据传输错误

下面是针对每个 Kinesis Data Firehose 目标的数据传输错误代码和消息的列表。每个错误消息还描述了修复问题所应采取的适当操作。

Amazon S3 数据传输错误

Kinesis Data Firehose 可以向 CloudWatch Logs 发送以下 Amazon S3 相关错误。

错误代码 错误消息和信息
S3.KMS.NotFoundException

"The provided AWS KMS key was not found.If you are using what you believe to be a valid AWS KMS key with the correct role, check if there is a problem with the account to which the AWS KMS key is attached."

S3.KMS.RequestLimitExceeded

"The KMS request per second limit was exceeded while attempting to encrypt S3 objects.Increase the request per second limit."

有关更多信息,请参阅 AWS Key Management Service Developer Guide 中的限制

S3.AccessDenied "Access was denied.Ensure that the trust policy for the provided IAM role allows Kinesis Data Firehose to assume the role, and the access policy allows access to the S3 bucket."
S3.AccountProblem "There is a problem with your AWS account that prevents the operation from completing successfully.Contact AWS Support."
S3.AllAccessDisabled "Access to the account provided has been disabled.Contact AWS Support."
S3.InvalidPayer "Access to the account provided has been disabled.Contact AWS Support."
S3.NotSignedUp "The account is not signed up for Amazon S3.Sign the account up or use a different account."
S3.NoSuchBucket "The specified bucket does not exist.Create the bucket or use a different bucket that does exist."
S3.MethodNotAllowed "The specified method is not allowed against this resource.Modify the bucket’s policy to allow the correct Amazon S3 operation permissions."
InternalError "An internal error occurred while attempting to deliver data.Delivery will be retried; if the error persists, then it will be reported to AWS for resolution."

Amazon Redshift 数据传输错误

Kinesis Data Firehose 可以向 CloudWatch Logs 发送以下 Amazon Redshift 相关错误。

错误代码 错误消息和信息
Redshift.TableNotFound

"The table to which to load data was not found.Ensure that the specified table exists."

The destination table in Amazon Redshift to which data should be copied from S3 was not found.Note that Kinesis Data Firehose does not create the Amazon Redshift table if it does not exist.

Redshift.SyntaxError "The COPY command contains a syntax error.Retry the command."
Redshift.AuthenticationFailed "The provided user name and password failed authentication.Provide a valid user name and password."
Redshift.AccessDenied "Access was denied.Ensure that the trust policy for the provided IAM role allows Kinesis Data Firehose to assume the role."
Redshift.S3BucketAccessDenied "The COPY command was unable to access the S3 bucket.Ensure that the access policy for the provided IAM role allows access to the S3 bucket."
Redshift.DataLoadFailed "Loading data into the table failed.Check STL_LOAD_ERRORS system table for details."
Redshift.ColumnNotFound "A column in the COPY command does not exist in the table.Specify a valid column name."
Redshift.DatabaseNotFound "The database specified in the Amazon Redshift destination configuration or JDBC URL was not found.Specify a valid database name."
Redshift.IncorrectCopyOptions

"Conflicting or redundant COPY options were provided.Some options are not compatible in certain combinations.Check the COPY command reference for more info."

有关更多信息,请参阅Amazon Redshift Database Developer Guide中的 Amazon Redshift COPY 命令

Redshift.MissingColumn "There is a column defined in the table schema as NOT NULL without a DEFAULT value and not included in the column list.Exclude this column, ensure that the loaded data always provides a value for this column, or add a default value to the Amazon Redshift schema for this table."
Redshift.ConnectionFailed "The connection to the specified Amazon Redshift cluster failed.Ensure that security settings allow Kinesis Data Firehose connections, that the cluster or database specified in the Amazon Redshift destination configuration or JDBC URL is correct, and that the cluster is available."
Redshift.ColumnMismatch "The number of jsonpaths in the COPY command and the number of columns in the destination table should match.Retry the command."
Redshift.IncorrectOrMissingRegion "Amazon Redshift attempted to use the wrong region endpoint for accessing the S3 bucket.Either specify a correct region value in the COPY command options or ensure that the S3 bucket is in the same region as the Amazon Redshift database."
Redshift.IncorrectJsonPathsFile "The provided jsonpaths file is not in a supported JSON format.Retry the command."
Redshift.MissingS3File "One or more S3 files required by Amazon Redshift have been removed from the S3 bucket.Check the S3 bucket policies to remove any automatic deletion of S3 files."
Redshift.InsufficientPrivilege "The user does not have permissions to load data into the table.Check the Amazon Redshift user permissions for the INSERT privilege."
Redshift.ReadOnlyCluster "The query cannot be executed because the system is in resize mode.Try the query again later."
Redshift.DiskFull "Data could not be loaded because the disk is full.Increase the capacity of the Amazon Redshift cluster or delete unused data to free disk space."
InternalError "An internal error occurred while attempting to deliver data.Delivery will be retried; if the error persists, then it will be reported to AWS for resolution."

Splunk 数据传输错误

Kinesis Data Firehose can send the following Splunk-related errors to CloudWatch Logs.

错误代码 错误消息和信息
Splunk.ProxyWithoutStickySessions

"If you have a proxy (ELB or other) between Kinesis Data Firehose and the HEC node, you must enable sticky sessions to support HEC ACKs."

Splunk.DisabledToken "The HEC token is disabled.Enable the token to allow data delivery to Splunk."
Splunk.InvalidToken "The HEC token is invalid.Update Kinesis Data Firehose with a valid HEC token."
Splunk.InvalidDataFormat "The data is not formatted correctly.To see how to properly format data for Raw or Event HEC endpoints, see Splunk Event Data."
Splunk.InvalidIndex "The HEC token or input is configured with an invalid index.Check your index configuration and try again."
Splunk.ServerError "Data delivery to Splunk failed due to a server error from the HEC node. Kinesis Data Firehose will retry sending the data if the retry duration in your Kinesis Data Firehose is greater than 0.If all the retries fail, Kinesis Data Firehose backs up the data to Amazon S3."
Splunk.DisabledAck "Indexer acknowledgement is disabled for the HEC token.Enable indexer acknowledgement and try again.For more info, see Enable indexer acknowledgement."
Splunk.AckTimeout "Did not receive an acknowledgement from HEC before the HEC acknowledgement timeout expired.Despite the acknowledgement timeout, it's possible the data was indexed successfully in Splunk. Kinesis Data Firehose backs up in Amazon S3 data for which the acknowledgement timeout expired."
Splunk.MaxRetriesFailed

"Failed to deliver data to Splunk or to receive acknowledgment.Check your HEC health and try again."

Splunk.ConnectionTimeout "The connection to Splunk timed out.This might be a transient error and the request will be retried. Kinesis Data Firehose backs up the data to Amazon S3 if all retries fail."
Splunk.InvalidEndpoint "Could not connect to the HEC endpoint.Make sure that the HEC endpoint URL is valid and reachable from Kinesis Data Firehose."
Splunk.ConnectionClosed "Unable to send data to Splunk due to a connection failure.This might be a transient error.Increasing the retry duration in your Kinesis Data Firehose configuration might guard against such transient failures."
Splunk.SSLUnverified "Could not connect to the HEC endpoint.主机与对等项提供的证书不匹配。请确保证书和主机是有效的。
Splunk.SSLHandshake "Could not connect to the HEC endpoint.请确保证书和主机是有效的。

Amazon Elasticsearch Service 数据传输错误

对于 Amazon ES 目标,Kinesis Data Firehose 在 Elasticsearch 返回错误时将它们发送到 CloudWatch Logs。

Lambda 调用错误

Kinesis Data Firehose 可以向 CloudWatch Logs 发送以下 Lambda 调用错误。

错误代码 错误消息和信息
Lambda.AssumeRoleAccessDenied

"Access was denied.Ensure that the trust policy for the provided IAM role allows Kinesis Data Firehose to assume the role."

Lambda.InvokeAccessDenied

"Access was denied.Ensure that the access policy allows access to the Lambda function."

Lambda.JsonProcessingException

"There was an error parsing returned records from the Lambda function.Ensure that the returned records follow the status model required by Kinesis Data Firehose."

有关更多信息,请参阅数据转换和状态模型

Lambda.InvokeLimitExceeded

"The Lambda concurrent execution limit is exceeded.Increase the concurrent execution limit."

有关更多信息,请参阅 AWS Lambda Developer Guide 中的 AWS Lambda 限制

Lambda.DuplicatedRecordId

"Multiple records were returned with the same record ID.Ensure that the Lambda function returns unique record IDs for each record."

有关更多信息,请参阅数据转换和状态模型

Lambda.MissingRecordId

"One or more record IDs were not returned.Ensure that the Lambda function returns all received record IDs."

有关更多信息,请参阅数据转换和状态模型

Lambda.ResourceNotFound

"The specified Lambda function does not exist.Use a different function that does exist."

Lambda.InvalidSubnetIDException

"The specified subnet ID in the Lambda function VPC configuration is invalid.Ensure that the subnet ID is valid."

Lambda.InvalidSecurityGroupIDException

"The specified security group ID in the Lambda function VPC configuration is invalid.Ensure that the security group ID is valid."

Lambda.SubnetIPAddressLimitReachedException

"AWS Lambda was not able to set up the VPC access for the Lambda function because one or more configured subnets have no available IP addresses.Increase the IP address limit."

有关更多信息,请参阅 Amazon VPC 用户指南 中的 Amazon VPC 限制 - VPC 和子网

Lambda.ENILimitReachedException

"AWS Lambda was not able to create an Elastic Network Interface (ENI) in the VPC, specified as part of the Lambda function configuration, because the limit for network interfaces has been reached.Increase the network interface limit."

有关更多信息,请参阅 Amazon VPC 用户指南 中的 Amazon VPC 限制 - 网络接口

访问 Kinesis Data Firehose 的 CloudWatch 日志

可以使用 Kinesis Data Firehose 控制台或 CloudWatch 控制台查看与 Kinesis Data Firehose 数据传输失败有关的错误日志。下面的过程介绍如何使用这两种方法访问错误日志。

使用 Kinesis Data Firehose 控制台访问错误日志

  1. 登录 AWS 管理控制台 并通过以下网址打开 Kinesis 控制台:https://console.amazonaws.cn/kinesis

  2. 在导航窗格中选择 Data Firehose (数据 Firehose)

  3. 在导航栏中,选择 AWS 区域。

  4. 选择传输流名称,转到传输流详细信息页面。

  5. 选择 Error Log 查看与数据传输故障有关的错误日志的列表。

使用 CloudWatch 控制台访问错误日志

  1. 通过以下网址打开 CloudWatch 控制台:https://console.amazonaws.cn/cloudwatch/

  2. 在导航栏中,选择一个区域。

  3. 在导航窗格中,选择 Logs

  4. 选择日志组和日志流,查看与数据传输故障有关的错误日志的列表。