Ingesting data using Amazon IoT Core rules - Amazon IoT SiteWise
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Ingesting data using Amazon IoT Core rules

You can send data to Amazon IoT SiteWise from Amazon IoT things and other Amazon services by using rules in Amazon IoT Core. Rules transform MQTT messages and perform actions to interact with Amazon services. The Amazon IoT SiteWise rule action forwards messages data to the BatchPutAssetPropertyValue operation from the Amazon IoT SiteWise API. For more information, see Rules and Amazon IoT SiteWise action in the Amazon IoT Developer Guide.

You can follow a tutorial that walks through the steps required to set up a rule that ingests data from Amazon IoT things through their device shadows. For more information, see Ingesting data to Amazon IoT SiteWise from Amazon IoT things.

You can also send data from Amazon IoT SiteWise to other Amazon services. For more information, see Interacting with other Amazon services.

Granting Amazon IoT the required access

You use IAM roles to control the Amazon resources to which each rule has access. Before you create a rule, you must create an IAM role with a policy that allows the rule to perform actions on the required Amazon resource. Amazon IoT assumes this role when running a rule.

If you create the rule action in the Amazon IoT console, you can choose a root asset to create a role that has access to a selected asset hierarchy. For more information about how to manually define a role for a rule, see Granting Amazon IoT the required access and Pass role permissions in the Amazon IoT Developer Guide.

For the Amazon IoT SiteWise rule action, you must define a role that allows iotsitewise:BatchPutAssetPropertyValue access to the asset properties to which the rule sends data. To improve security, you can specify an Amazon IoT SiteWise asset hierarchy path in the Condition property.

The following example trust policy allows access to a specific asset and its children.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iotsitewise:BatchPutAssetPropertyValue", "Resource": "*", "Condition": { "StringLike": { "iotsitewise:assetHierarchyPath": [ "/root node asset ID", "/root node asset ID/*" ] } } } ] }

You can remove the Condition from the policy to allow access to all of your assets. The following example trust policy allows access to all of your assets.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iotsitewise:BatchPutAssetPropertyValue", "Resource": "*" } ] }

Configuring the Amazon IoT SiteWise rule action

The Amazon IoT SiteWise rule action sends data from the MQTT message that initiated the rule to asset properties in Amazon IoT SiteWise. You can upload multiple data entries to different asset properties at the same time, so that you can send updates for all sensors of a device in one message. You can also upload multiple data points at once for each data entry.

Note

When you send data to Amazon IoT SiteWise with the rule action, your data must meet all of the requirements of the BatchPutAssetPropertyValue operation. For example, your data can't have a timestamp earlier than 7 days from current Unix epoch time. For more information, see Ingesting data with the Amazon IoT SiteWise API.

For each data entry in the rule action, you identify an asset property and specify the timestamp, quality, and value of each data point for that asset property. The rule action expects strings for all parameters.

To identify an asset property in an entry, specify one of the following:

  • The Asset ID (assetId) and Property ID (propertyId) of the asset property that you're sending data to. You can find the Asset ID and Property ID using the Amazon IoT SiteWise console. If you know the Asset ID, you can use the Amazon CLI to call DescribeAsset to find the Property ID.

  • The Property alias (propertyAlias), which is a data stream alias (for example, /company/windfarm/3/turbine/7/temperature). To use this option, you must first set your asset property's alias. To learn how to set property aliases, see Mapping industrial data streams to asset properties.

For the timestamp in each entry, you can use the timestamp reported by your equipment or the timestamp provided by Amazon IoT Core. The timestamp has two parameters:

  • Time in seconds (timeInSeconds) – The Unix epoch time, in seconds, at which the sensor or equipment reported the data.

  • Offset in nanos (offsetInNanos) – (Optional) The nanosecond offset from the time in seconds.

Important

If your timestamp is a string, has a decimal portion, or isn't in seconds, Amazon IoT SiteWise rejects the request. You must convert the timestamp to seconds and nanosecond offset. Use features of the Amazon IoT rules engine to convert the timestamp. For more information, see the following:

You can use substitution templates for several parameters in the action to perform calculations, invoke functions, and pull values from the message payload. For more information, see Substitution templates in the Amazon IoT Developer Guide.

Note

Because an expression in a substitution template is evaluated separately from the SELECT statement, you can't use a substitution template to reference an alias created using an AS clause. You can reference only information present in the original payload, in addition to supported functions and operators.

Getting timestamps for devices that don't report accurate time

If your sensor or equipment doesn't report accurate time data, you can get the current Unix epoch time from the Amazon IoT rules engine with timestamp(). This function outputs time in milliseconds, so you must convert the value to time in seconds and offset in nanoseconds. To do so, use the following conversions:

  • For Time in seconds (timeInSeconds), use ${floor(timestamp() / 1E3)} to convert the time from milliseconds to seconds.

  • For Offset in nanos (offsetInNanos), use ${(timestamp() % 1E3) * 1E6} to calculate the nanosecond offset of the timestamp.

Converting timestamps that are in string format

If your sensor or equipment reports time data in string format (for example, 2020-03-03T14:57:14.699Z), you can use time_to_epoch(String, String). This function inputs the timestamp and format pattern as parameters and outputs time in milliseconds. Then, you must convert the time to time in seconds and offset in nanoseconds. To do so, use the following conversions:

  • For Time in seconds (timeInSeconds), use ${floor(time_to_epoch("2020-03-03T14:57:14.699Z", "yyyy-MM-dd'T'HH:mm:ss'Z'") / 1E3)} to convert the timestamp string to milliseconds, and then to seconds.

  • For Offset in nanos (offsetInNanos), use ${(time_to_epoch("2020-03-03T14:57:14.699Z", "yyyy-MM-dd'T'HH:mm:ss'Z'") % 1E3) * 1E6} to calculate the nanosecond offset of the timestamp string.

Note

The time_to_epoch function supports up to millisecond-precision timestamp strings. To convert strings with microsecond or nanosecond precision, you can configure an Amazon Lambda function that your rule calls to convert the timestamp into numerical values. For more information, see Converting nanosecond-precision timestamp strings.

Converting nanosecond-precision timestamp strings

If your device sends timestamp information in string format with nanosecond precision (for example, 2020-03-03T14:57:14.699728491Z), use the following procedure to configure your rule action. You can create an Amazon Lambda function that converts the timestamp from a string into Time in seconds (timeInSeconds) and Offset in nanos (offsetInNanos). Then, you can use aws_lambda(functionArn, inputJson) in your rule action parameters to invoke that Lambda function and use the output in your rule.

Note

This section contains advanced instructions that assume that you're familiar with how to create the following resources:

To create an Amazon IoT SiteWise rule action that parses timestamp strings
  1. Create a Lambda function with the following properties:

    • Function name – Use a descriptive function name (for example, ConvertNanosecondTimestampFromString).

    • Runtime – Use a Python 3 runtime, such as Python 3.11 (python3.11).

    • Permissions – Create a role with basic Lambda permissions (AmazonLambdaBasicExecutionRole).

    • Layers – Add the AmazonSDKPandas-Python311 layer for the Lambda function to use numpy.

    • Function code – Use the following function code, which consumes a string argument named timestamp and outputs timeInSeconds and offsetInNanos values for that timestamp.

      import json import math import numpy # Converts a timestamp string into timeInSeconds and offsetInNanos in Unix epoch time. # The input timestamp string can have up to nanosecond precision. def lambda_handler(event, context): timestamp_str = event['timestamp'] # Parse the timestamp string as nanoseconds since Unix epoch. nanoseconds = numpy.datetime64(timestamp_str, 'ns').item() time_in_seconds = math.floor(nanoseconds / 1E9) # Slice to avoid precision issues. offset_in_nanos = int(str(nanoseconds)[-9:]) return { 'timeInSeconds': time_in_seconds, 'offsetInNanos': offset_in_nanos }

      This Lambda function inputs timestamp strings in ISO 8601 format using datetime64 from NumPy.

      Note

      If your timestamp strings aren't in ISO 8601 format, you can implement a solution with pandas that defines the timestamp format. For more information, see pandas.to_datetime.

  2. When you configure the Amazon IoT SiteWise action for your rule, use the following substitution templates for Time in seconds (timeInSeconds) and Offset in nanos (offsetInNanos). These substitution templates assume that your message payload contains the timestamp string in timestamp. The aws_lambda function consumes a JSON structure for its second parameter, so you can modify the below substitution templates if needed.

    • For Time in seconds (timeInSeconds), use the following substitution template.

      ${aws_lambda('arn:aws:lambda:region:account-id:function:ConvertNanosecondTimestampFromString', {'timestamp': timestamp}).timeInSeconds}
    • For Offset in nanos (offsetInNanos), use the following substitution template.

      ${aws_lambda('arn:aws:lambda:region:account-id:function:ConvertNanosecondTimestampFromString', {'timestamp': timestamp}).offsetInNanos}

    For each parameter, replace region and account-id with your Region and Amazon account ID. If you used a different name for your Lambda function, change that as well.

  3. Grant Amazon IoT permissions to invoke your function with the lambda:InvokeFunction permission. For more information, see aws_lambda(functionArn, inputJson).

  4. Test your rule (for example, use the Amazon IoT MQTT test client) and verify that Amazon IoT SiteWise receives the data that you send.

    If your rule doesn't work as expected, see Troubleshooting an Amazon IoT SiteWise rule action.

Note

This solution invokes the Lambda function twice for each timestamp string. You can create another rule to reduce the number of Lambda function invocations if your rule handles multiple data points that have the same timestamp in each payload.

To do so, create a rule with a republish action that invokes the Lambda and publishes the original payload with the timestamp string converted to timeInSeconds and offsetInNanos. Then, create a rule with an Amazon IoT SiteWise rule action to consume the converted payload. With this approach, you reduce the number of times that the rule invokes the Lambda but increase the number of Amazon IoT rule actions run. Consider the pricing of each service if you apply this solution to your use case.

Example rule configurations

This section contains example rule configurations that you can use to create a rule with an Amazon IoT SiteWise action.

Example rule action that uses property aliases as message topics

The following example creates a rule with an Amazon IoT SiteWise action that uses the topic (through topic()) as the property alias to identify asset properties. You can use this example to define one rule for ingesting double-type data to all wind turbines in all wind farms. This example requires that you define property aliases on all turbine assets' properties. You would need to define a second, similar rule to ingest integer-type data.

aws iot create-topic-rule \ --rule-name SiteWiseWindFarmRule \ --topic-rule-payload file://sitewise-rule-payload.json

The example payload in sitewise-rule-payload.json contains the following content.

{ "sql": "SELECT * FROM '/company/windfarm/+/turbine/+/+' WHERE type = 'double'", "description": "Sends data to the wind turbine asset property with the same alias as the topic", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [ { "iotSiteWise": { "putAssetPropertyValueEntries": [ { "propertyAlias": "${topic()}", "propertyValues": [ { "timestamp": { "timeInSeconds": "${timeInSeconds}" }, "value": { "doubleValue": "${value}" } } ] } ], "roleArn": "arn:aws:iam::account-id:role/MySiteWiseActionRole" } } ] }

With this rule action, you can send the following message to a wind turbine property alias (for example, /company/windfarm/3/turbine/7/temperature) as a topic to ingest data.

{ "type": "double", "value": "38.3", "timeInSeconds": "1581368533" }
Example rule action that uses timestamp() to determine time

The following example creates a rule with an Amazon IoT SiteWise action that identifies an asset property by IDs and uses timestamp() to determine the current time.

aws iot create-topic-rule \ --rule-name SiteWiseAssetPropertyRule \ --topic-rule-payload file://sitewise-rule-payload.json

The example payload in sitewise-rule-payload.json contains the following content.

{ "sql": "SELECT * FROM 'my/asset/property/topic'", "description": "Sends device data to an asset property", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [ { "iotSiteWise": { "putAssetPropertyValueEntries": [ { "assetId": "a1b2c3d4-5678-90ab-cdef-22222EXAMPLE", "propertyId": "a1b2c3d4-5678-90ab-cdef-33333EXAMPLE", "propertyValues": [ { "timestamp": { "timeInSeconds": "${floor(timestamp() / 1E3)}", "offsetInNanos": "${(timestamp() % 1E3) * 1E6}" }, "value": { "doubleValue": "${value}" } } ] } ], "roleArn": "arn:aws:iam::account-id:role/MySiteWiseActionRole" } } ] }

With this rule action, you can send the following message to the my/asset/property/topic to ingest data.

{ "type": "double", "value": "38.3" }

Reducing costs with basic ingest

Amazon IoT Core provides a feature called Basic Ingest that you can use to send data through Amazon IoT Core without incurring Amazon IoT messaging costs. Basic Ingest optimizes data flow for high volume data ingestion workloads by removing the publish/subscribe message broker from the ingestion path. You can use Basic Ingest if you know which rules your messages should be routed to.

To use Basic Ingest, you send messages directly to a specific rule using a special topic, $aws/rules/rule-name. For example, to send a message to a rule named SiteWiseWindFarmRule, you send a message to the topic $aws/rules/SiteWiseWindFarmRule.

If your rule action uses substitution templates that contain topic(Decimal), you can pass the original topic at the end of the Basic Ingest special topic, such as $aws/rules/rule-name/original-topic. For example, to use Basic Ingest with the wind farm property alias example from the previous section, you can send messages to the following topic.

$aws/rules/SiteWiseWindFarmRule//company/windfarm/3/turbine/7/temperature
Note

The above example includes a second slash (//) because Amazon IoT removes the Basic Ingest prefix ($aws/rules/rule-name/) from the topic that's visible to the rule action. In this example, the rule receives the topic /company/windfarm/3/turbine/7/temperature.

For more information, see Reducing messaging costs with basic ingest in the Amazon IoT Developer Guide.

Troubleshooting the Amazon IoT SiteWise rule action

To troubleshoot your Amazon IoT SiteWise rule action in Amazon IoT Core, you can configure CloudWatch Logs or you can configure a republish error action for your rule. For more information, see Troubleshooting an Amazon IoT SiteWise rule action.