Gathering system health telemetry data from Amazon IoT Greengrass core devices - Amazon IoT Greengrass
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Gathering system health telemetry data from Amazon IoT Greengrass core devices

System health telemetry data is diagnostic data that can help you monitor the performance of critical operations on your Greengrass core devices. The telemetry agent on the Greengrass core collects local telemetry data and publishes it to Amazon EventBridge without requiring any customer interaction. Core devices publish telemetry data to EventBridge on a best effort basis. For example, core devices might fail to deliver telemetry data while offline.

Note

Amazon EventBridge is an event bus service that you can use to connect your applications with data from a variety of sources, such as Greengrass core devices and deployment notifications. For more information, see What is Amazon EventBridge? in the Amazon EventBridge User Guide.

You can create projects and applications to retrieve, analyze, transform, and report telemetry data from your edge devices. Domain experts, such as process engineers, can use these applications to gain insights into fleet health.

To ensure that the Greengrass edge components function properly, Amazon IoT Greengrass uses the data for development and quality improvement purposes. This feature also helps inform new and enhanced edge capabilities. Amazon IoT Greengrass only retains telemetry data for up to seven days.

This feature is available in Amazon IoT Greengrass Core software v1.11.0 and is enabled by default for all Greengrass cores, including existing cores. You automatically start receiving data as soon as you upgrade to Amazon IoT Greengrass Core software v1.11.0 or later.

For information about how to access or manage published telemetry data, see Subscribing to receive telemetry data.

The telemetry agent collects and publishes the following system metrics.

Telemetry metrics
Name Description Source

SystemMemUsage

The amount of memory currently in use by all applications on the Greengrass core device, including the operating system.

System

CpuUsage

The amount of CPU currently in use by all applications on the Greengrass core device, including the operating system.

System

TotalNumberOfFDs

The number of file descriptors stored by the operating system of the Greengrass core device. One file descriptor uniquely identifies one open file.

System

LambdaOutOfMemory

The number of runs that result in the Lambda function running out of memory.

System

DroppedMessageCount

The number of dropped messages that are destined for Amazon IoT Core.

GGCloudSpooler system component

LambdaTimeout

The number of timeouts for running the user-defined Lambda function.

User-defined Lambda function, Amazon Web Services Cloud, and system

LambdaUngracefullyKilled

The number of runs that the user-defined Lambda function fails to complete.

User-defined Lambda function, Amazon Web Services Cloud, and system

LambdaError

The number of runs that result in the user-defined Lambda function writing error logs.

User-defined Lambda function, Amazon Web Services Cloud, and system

BytesAppended

The number of bytes of data appended to stream manager.

GGStreamManager system component

BytesUploadedToIoTAnalytics

The number of bytes of data that stream manager exports to channels in Amazon IoT Analytics.

GGStreamManager system component

BytesUploadedToKinesis

The number of bytes of data that stream manager exports to streams in Amazon Kinesis Data Streams.

GGStreamManager system component

BytesUploadedToIoTSiteWise

The number of bytes of data that stream manager exports to asset properties in Amazon IoT SiteWise.

GGStreamManager system component

BytesUploadedToS3ExportTaskExecutor

The number of bytes of data that stream manager exports to objects in Amazon S3.

GGStreamManager system component

BytesUploadedToHTTP

The number of bytes of data that stream manager exports to HTTP.

GGStreamManager system component

Configuring telemetry settings

Greengrass telemetry uses the following settings:

  • The telemetry agent aggregates telemetry data every hour.

  • The telemetry agent publishes a telemetry message every 24 hours.

Note

The settings are unchangeable.

You can enable or disable the telemetry feature for a Greengrass core device. Amazon IoT Greengrass uses shadows to manage the telemetry configuration. Your changes take effect immediately when the core has a connection to Amazon IoT Core.

The telemetry agent publishes data using the MQTT protocol with a quality of service (QoS) level of 0. This means that it doesn't confirm delivery or retry publishing attempts. Telemetry messages share an MQTT connection with other messages for subscriptions destined for Amazon IoT Core.

Aside from your data link costs, the data transfer from the core to Amazon IoT Core is no charge. This is because the agent publishes to an Amazon reserved topic. However, depending on your use case, you might incur costs when you receive or process the data.

Requirements

The following requirements apply, when you configure telemetry settings:

  • You must use Amazon IoT Greengrass Core software v1.11.0 or later.

    Note

    If you're running an earlier version and you don't want to use telemetry, you don't have to do anything.

  • You must provide IAM permissions to update the core (thing) shadow and to call the configuration APIs before you update telemetry settings.

    The following example IAM policy lets you manage the shadow and runtime configuration of a specific core:

    { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowManageShadow", "Effect": "Allow", "Action": [ "iot:GetThingShadow", "iot:UpdateThingShadow", "iot:DeleteThingShadow", "iot:DescribeThing" ], "Resource": [ "arn:aws-cn:iot:region:account-id:thing/core-name-*" ] }, { "Sid": "AllowManageRuntimeConfig", "Effect": "Allow", "Action": [ "greengrass:GetCoreRuntimeConfiguration", "greengrass:UpdateCoreRuntimeConfiguration" ], "Resource": [ "arn:aws-cn:iot:region:account-id:thing/core-name" ] } ] }

    You can grant granular or conditional access to resources, for example, by using a wildcard * naming scheme. For more information, see Adding and removing IAM policies in the IAM User Guide.

Configure telemetry settings (console)

The following shows how to update the telemetry settings of a Greengrass core in the Amazon IoT Greengrass console.

  1. In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).

  2. Under Greengrass groups, choose your target group.

  3. On the group configuration page, in the Overview section, choose your Greengrass core.

  4. On the core's configuration page, choose the Telemetry tab.

  5. In the System health telemetry section, choose Configure.

  6. In Configure telemetry, select Telemetry to enable or disable the Telemetry status.

    Important

    By default, the telemetry feature is enabled for Amazon IoT Greengrass Core software v1.11.0 or later.

The changes take effect at runtime. You don't need to deploy the group.

Configure telemetry settings (CLI)

In the Amazon IoT Greengrass API, the TelemetryConfiguration object represents the telemetry settings of a Greengrass core. This object is part of the RuntimeConfiguration object associated with the core. You can use the Amazon IoT Greengrass API, Amazon CLI, or Amazon SDK to manage Greengrass telemetry. The examples in this section use the Amazon CLI.

To check telemetry settings

The following command gets the telemetry settings of a Greengrass core.

  • Replace core-thing-name with the name of the target core.

    To get the thing name, you use the get-core-definition-version command. The command returns the ARN of the thing that contains the thing name.

aws greengrass get-thing-runtime-configuration --thing-name core-thing-name

The command returns a GetCoreRuntimeConfigurationResponse object in the JSON response. For example:

{ "RuntimeConfiguration": { "TelemetryConfiguration": { "ConfigurationSyncStatus": "OutOfSync", "Telemetry": "On" } } }
To configure telemetry settings

The following command updates the telemetry settings for a Greengrass core.

  • Replace core-thing-name with the name of the target core.

    To get the thing name, you use the get-core-definition-version command. The command returns the ARN of the thing that contains the thing name.

JSON expanded
aws greengrass update-thing-runtime-configuration --thing-name core-thing-name --telemetry-configuration '{ "RuntimeConfiguration": { "TelemetryConfiguration": { "ConfigurationSyncStatus": "InSync", "Telemetry": "Off" } } }
JSON single-line
aws greengrass update-thing-runtime-configuration --thing-name core-thing-name --telemetry-configuration "{\"TelemetryConfiguration\":{\"ConfigurationSyncStatus\":\"InSync\",\"Telemetry\":\"Off\"}}"

Changes to telemetry settings have been applied if the ConfigurationSyncStatus is InSync. The changes take effect at runtime. You don't need to deploy the group.

TelemetryConfiguration object

The TelemetryConfiguration object has the following properties:

ConfigurationSyncStatus

Checks if telemetry settings are in sync. You might not make changes to this property.

Type: string

Valid values: InSync or OutOfSync

Telemetry

Turns telemetry on or off. The default is On.

Type: string

Valid values: On or Off

Subscribing to receive telemetry data

You can create rules in Amazon EventBridge that define how to process telemetry data published from the Greengrass core device. When EventBridge receives the data, it invokes the target actions defined in your rules. For example, you can create event rules that send notifications, store event information, take corrective action, or invoke other events.

Telemetry event

The event for a deployment state change including the telemetry data uses the following format:

{ "version": "0", "id": "f70f943b-9ae2-e7a5-fec4-4c22178a3e6a", "detail-type": "Greengrass Telemetry Data", "source": "aws.greengrass", "account": "123456789012", "time": "2020-07-28T20:45:53Z", "region": "us-west-1", "resources": [], "detail": { "ThingName": "CoolThing", "Schema": "2020-06-30", "ADP": [ { "TS": 123231546, "NS": "StreamManager", "M": [ { "N": "BytesAppended|BytesUploadedToKinesis", "Sum": 11, "U": "Bytes" } ] }, { "TS": 123231546, "NS": "StreamManager", "M": [ { "N": "BytesAppended|BytesUploadedToS3ExportTaskExecutor", "Sum": 11, "U": "Bytes" } ] }, { "TS": 123231546, "NS": "StreamManager", "M": [ { "N": "BytesAppended|BytesUploadedToHTTP", "Sum": 11, "U": "Bytes" } ] }, { "TS": 123231546, "NS": "StreamManager", "M": [ { "N": "BytesAppended|BytesUploadedToIoTAnalytics", "Sum": 11, "U": "Bytes" } ] }, { "TS": 123231546, "NS": "StreamManager", "M": [ { "N": "BytesAppended|BytesUploadedToIoTSiteWise", "Sum": 11, "U": "Bytes" } ] }, { "TS": 123231546, "NS": "arn:aws:lambda:us-west-1:123456789012:function:my-function", "M": [ { "N": "LambdaTimeout", "Sum": 15, "U": "Count" } ] }, { "TS": 123231546, "NS": "CloudSpooler", "M": [ { "N": "DroppedMessageCount", "Sum": 15, "U": "Count" } ] }, { "TS": 1593727692, "NS": "SystemMetrics", "M": [ { "N": "SystemMemUsage", "Sum": 11.23, "U": "Megabytes" }, { "N": "CpuUsage", "Sum": 35.63, "U": "Percent" }, { "N": "TotalNumberOfFDs", "Sum": 416, "U": "Count" } ] }, { "TS": 1593727692, "NS": "arn:aws:lambda:us-west-1:123456789012:function:my-function", "M": [ { "N": "LambdaOutOfMemory", "Sum": 12, "U": "Count" }, { "N": "LambdaUngracefullyKilled", "Sum": 100, "U": "Count" }, { "N": "LambdaError", "Sum": 7, "U": "Count" } ] } ] } }

The ADP array contains a list of aggregated data points that have the following properties:

TS

Required. The timestamp of when the data was aggregated.

NS

Required. The namespace of the system.

M

Required. The list of metrics. A metric contains the following properties:

N

The name of the metric.

Sum

The aggregated metric value. The telemetry agent adds new values to the previous total, so the sum is an ever-increasing value. You can use the timestamp to find the value of a specific aggregation. For example, to find the latest aggregated value, subtract the previous timestamped value from the latest timestamped value.

U

The unit of the metric value.

ThingName

Required. The name of the thing device that you target.

Prerequisites for creating EventBridge rules

Before you create an EventBridge rule for Amazon IoT Greengrass, you should do the following:

  • Familiarize yourself with events, rules, and targets in EventBridge.

  • Create and configure the targets invoked by your EventBridge rules. Rules can invoke many types of targets, such as Amazon Kinesis streams, Amazon Lambda functions, Amazon SNS topics, and Amazon SQS queues.

    Your EventBridge rule, and the associated targets must be in the Amazon Web Services Region where you created your Greengrass resources. For more information, see Service endpoints and quotas in the Amazon Web Services General Reference.

For more information, see What is Amazon EventBridge? and Getting started with Amazon EventBridge in the Amazon EventBridge User Guide.

Create an event rule to get telemetry data (console)

Use the following steps to use the Amazon Web Services Management Console to create an EventBridge rule that receives telemetry data published by the Greengrass core. This allows web servers, email addresses, and other topic subscribers to respond to the event. For more information, see Creating a EventBridge rule that triggers on an event from an Amazon resource in the Amazon EventBridge User Guide.

  1. Open the Amazon EventBridge console and choose Create rule.

  2. Under Name and description, enter a name and description for the rule.

  3. Choose Event bus- and enable the rule on the selected event bus..

  4. Select the Rule type and choose Rule with an event pattern.

  5. Choose Next.

  6. For Event source, choose Amazon events or EventBridge partner events.

  7. For Sample event, choose Amazon events, and select Greengrass Telemetry Data.

  8. In Event pattern, make the following selections:

    1. For Event source, choose Amazon services.

    2. For Amazon service, choose Greengrass.

    3. For Event type, choose Greengrass Telemetry Data.

  9. Choose Next.

  10. For Target 1, choose Amazon service.

  11. For Select a target, choose SQS queue.

  12. For Queue, choose your function.

Create an event rule to get telemetry data (CLI)

Use the following steps to use the Amazon CLI to create an EventBridge rule that receives telemetry data published by the Greengrass core. This allows web servers, email addresses, and other topic subscribers to respond to the event.

  1. Create the rule.

    • Replace thing-name with the thing name of the core.

      To get the thing name, you use the get-core-definition-version command. The command returns the ARN of the thing that contains the thing name.

    aws events put-rule \ --name TestRule \ --event-pattern "{\"source\": [\"aws.greengrass\"], \"detail\": {\"ThingName\": [\"thing-name\"]}}"

    Properties that are omitted from the pattern are ignored.

  2. Add the topic as a rule target. The following example uses Amazon SQS but you can configure other target types.

    • Replace queue-arn with the ARN of your Amazon SQS queue.

    aws events put-targets \ --rule TestRule \ --targets "Id"="1","Arn"="queue-arn"
    Note

    To allow Amazon EventBridge to invoke your target queue, you must add a resource-based policy to your topic. For more information, see Amazon SQS permissions in the Amazon EventBridge User Guide.

For more information, see Events and event patterns in EventBridge in the Amazon EventBridge User Guide.

Troubleshooting Amazon IoT Greengrass telemetry

Use the following information to help troubleshoot issues with configuring Amazon IoT Greengrass telemetry.

Error: The response contains "ConfigurationStatus": "OutOfSync" after you run the get-thing-runtime-configuration command

Solutions:

  • The Amazon IoT Device Shadow service takes time to process runtime configuration updates and to deliver the updates to the Greengrass core device. You might wait and check if telemetry settings are in sync later.

  • Make sure that your core device is online.

  • Enable Amazon CloudWatch Logs in Amazon IoT Core to monitor the shadow.

  • Use Amazon IoT metrics to monitor your thing.