Ingesting data using Amazon IoT Core rules
You can send data to Amazon IoT SiteWise from Amazon IoT things and other Amazon services by using rules in Amazon IoT Core. Rules transform MQTT messages and perform actions to interact with Amazon services. The Amazon IoT SiteWise rule action forwards messages data to the BatchPutAssetPropertyValue operation from the Amazon IoT SiteWise API. For more information, see Rules and Amazon IoT SiteWise action in the Amazon IoT Developer Guide.
You can follow a tutorial that walks through the steps required to set up a rule that ingests data from Amazon IoT things through their device shadows. For more information, see Ingesting data to Amazon IoT SiteWise from Amazon IoT things.
You can also send data from Amazon IoT SiteWise to other Amazon services. For more information, see Interacting with other Amazon services.
Topics
Granting Amazon IoT the required access
You use IAM roles to control the Amazon resources to which each rule has access. Before you create a rule, you must create an IAM role with a policy that allows the rule to perform actions on the required Amazon resource. Amazon IoT assumes this role when running a rule.
If you create the rule action in the Amazon IoT console, you can choose a root asset to create a role that has access to a selected asset hierarchy. For more information about how to manually define a role for a rule, see Granting Amazon IoT the required access and Pass role permissions in the Amazon IoT Developer Guide.
For the Amazon IoT SiteWise rule action, you must define a role that allows
iotsitewise:BatchPutAssetPropertyValue
access to the asset properties to
which the rule sends data. To improve security, you can specify an Amazon IoT SiteWise asset hierarchy
path in the Condition
property.
The following example trust policy allows access to a specific asset and its children.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iotsitewise:BatchPutAssetPropertyValue", "Resource": "*", "Condition": { "StringLike": { "iotsitewise:assetHierarchyPath": [ "/
root node asset ID
", "/root node asset ID
/*" ] } } } ] }
You can remove the Condition
from the policy to allow access to all of your
assets. The following example trust policy allows access to all of your assets.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iotsitewise:BatchPutAssetPropertyValue", "Resource": "*" } ] }
Configuring the Amazon IoT SiteWise rule action
The Amazon IoT SiteWise rule action sends data from the MQTT message that initiated the rule to asset properties in Amazon IoT SiteWise. You can upload multiple data entries to different asset properties at the same time, so that you can send updates for all sensors of a device in one message. You can also upload multiple data points at once for each data entry.
Note
When you send data to Amazon IoT SiteWise with the rule action, your data must meet all of the
requirements of the BatchPutAssetPropertyValue
operation. For example, your
data can't have a timestamp earlier than 7 days from current Unix epoch time. For more
information, see Ingesting data with the Amazon IoT SiteWise
API.
For each data entry in the rule action, you identify an asset property and specify the timestamp, quality, and value of each data point for that asset property. The rule action expects strings for all parameters.
To identify an asset property in an entry, specify one of the following:
-
The Asset ID (
assetId
) and Property ID (propertyId
) of the asset property that you're sending data to. You can find the Asset ID and Property ID using the Amazon IoT SiteWise console. If you know the Asset ID, you can use the Amazon CLI to call DescribeAsset to find the Property ID. -
The Property alias (
propertyAlias
), which is a data stream alias (for example,/company/windfarm/3/turbine/7/temperature
). To use this option, you must first set your asset property's alias. To learn how to set property aliases, see Mapping industrial data streams to asset properties.
For the timestamp in each entry, you can use the timestamp reported by your equipment or the timestamp provided by Amazon IoT Core. The timestamp has two parameters:
-
Time in seconds (
timeInSeconds
) – The Unix epoch time, in seconds, at which the sensor or equipment reported the data. -
Offset in nanos (
offsetInNanos
) – (Optional) The nanosecond offset from the time in seconds.
Important
If your timestamp is a string, has a decimal portion, or isn't in seconds, Amazon IoT SiteWise rejects the request. You must convert the timestamp to seconds and nanosecond offset. Use features of the Amazon IoT rules engine to convert the timestamp. For more information, see the following:
You can use substitution templates for several parameters in the action to perform calculations, invoke functions, and pull values from the message payload. For more information, see Substitution templates in the Amazon IoT Developer Guide.
Note
Because an expression in a substitution template is evaluated separately from the
SELECT
statement, you can't use a substitution template to reference an
alias created using an AS
clause. You can reference only information present
in the original payload, in addition to supported functions and operators.
Topics
Getting timestamps for devices that don't report accurate time
If your sensor or equipment doesn't report accurate time data, you can get the current Unix epoch time from the Amazon IoT rules engine with timestamp(). This function outputs time in milliseconds, so you must convert the value to time in seconds and offset in nanoseconds. To do so, use the following conversions:
-
For Time in seconds (
timeInSeconds
), use${floor(timestamp() / 1E3)}
to convert the time from milliseconds to seconds. -
For Offset in nanos (
offsetInNanos
), use${(timestamp() % 1E3) * 1E6}
to calculate the nanosecond offset of the timestamp.
Converting timestamps that are in string format
If your sensor or equipment reports time data in string format (for example,
2020-03-03T14:57:14.699Z
), you can use time_to_epoch(String, String). This function inputs the timestamp and format
pattern as parameters and outputs time in milliseconds. Then, you must convert the time to
time in seconds and offset in nanoseconds. To do so, use the following conversions:
-
For Time in seconds (
timeInSeconds
), use${floor(time_to_epoch("2020-03-03T14:57:14.699Z", "yyyy-MM-dd'T'HH:mm:ss'Z'") / 1E3)}
to convert the timestamp string to milliseconds, and then to seconds. -
For Offset in nanos (
offsetInNanos
), use${(time_to_epoch("2020-03-03T14:57:14.699Z", "yyyy-MM-dd'T'HH:mm:ss'Z'") % 1E3) * 1E6}
to calculate the nanosecond offset of the timestamp string.
Note
The time_to_epoch
function supports up to millisecond-precision
timestamp strings. To convert strings with microsecond or nanosecond precision, you can
configure an Amazon Lambda function that your rule calls to convert the timestamp into
numerical values. For more information, see Converting nanosecond-precision
timestamp strings.
Converting nanosecond-precision timestamp strings
If your device sends timestamp information in string format with nanosecond precision
(for example, 2020-03-03T14:57:14.699728491Z
), use the following procedure to
configure your rule action. You can create an Amazon Lambda function that converts the
timestamp from a string into Time in seconds
(timeInSeconds
) and Offset in nanos
(offsetInNanos
). Then, you can use aws_lambda(functionArn, inputJson) in your rule action parameters to invoke
that Lambda function and use the output in your rule.
Note
This section contains advanced instructions that assume that you're familiar with how to create the following resources:
-
Lambda functions. For more information, see Create a Lambda function with the console or Using Lambda with the Amazon CLI in the Amazon Lambda Developer Guide.
-
Amazon IoT rules with the Amazon IoT SiteWise rule action. For more information, see Ingesting data using Amazon IoT Core rules.
To create an Amazon IoT SiteWise rule action that parses timestamp strings
-
Create a Lambda function with the following properties:
-
Function name – Use a descriptive function name (for example,
ConvertNanosecondTimestampFromString
). -
Runtime – Use a Python 3 runtime, such as Python 3.11 (
python3.11
). -
Permissions – Create a role with basic Lambda permissions (AmazonLambdaBasicExecutionRole).
-
Layers – Add the AmazonSDKPandas-Python311 layer for the Lambda function to use
numpy
. -
Function code – Use the following function code, which consumes a string argument named
timestamp
and outputstimeInSeconds
andoffsetInNanos
values for that timestamp.import json import math import numpy # Converts a timestamp string into timeInSeconds and offsetInNanos in Unix epoch time. # The input timestamp string can have up to nanosecond precision. def lambda_handler(event, context): timestamp_str = event['timestamp'] # Parse the timestamp string as nanoseconds since Unix epoch. nanoseconds = numpy.datetime64(timestamp_str, 'ns').item() time_in_seconds = math.floor(nanoseconds / 1E9) # Slice to avoid precision issues. offset_in_nanos = int(str(nanoseconds)[-9:]) return { 'timeInSeconds': time_in_seconds, 'offsetInNanos': offset_in_nanos }
This Lambda function inputs timestamp strings in ISO 8601
format using datetime64 from NumPy. Note
If your timestamp strings aren't in ISO 8601 format, you can implement a solution with pandas that defines the timestamp format. For more information, see pandas.to_datetime
.
-
-
When you configure the Amazon IoT SiteWise action for your rule, use the following substitution templates for Time in seconds (
timeInSeconds
) and Offset in nanos (offsetInNanos
). These substitution templates assume that your message payload contains the timestamp string intimestamp
. Theaws_lambda
function consumes a JSON structure for its second parameter, so you can modify the below substitution templates if needed.-
For Time in seconds (
timeInSeconds
), use the following substitution template.${aws_lambda('arn:aws:lambda:
region
:account-id
:function:ConvertNanosecondTimestampFromString
', {'timestamp': timestamp}).timeInSeconds} -
For Offset in nanos (
offsetInNanos
), use the following substitution template.${aws_lambda('arn:aws:lambda:
region
:account-id
:function:ConvertNanosecondTimestampFromString
', {'timestamp': timestamp}).offsetInNanos}
For each parameter, replace
region
andaccount-id
with your Region and Amazon account ID. If you used a different name for your Lambda function, change that as well. -
-
Grant Amazon IoT permissions to invoke your function with the
lambda:InvokeFunction
permission. For more information, see aws_lambda(functionArn, inputJson). -
Test your rule (for example, use the Amazon IoT MQTT test client) and verify that Amazon IoT SiteWise receives the data that you send.
If your rule doesn't work as expected, see Troubleshooting an Amazon IoT SiteWise rule action.
Note
This solution invokes the Lambda function twice for each timestamp string. You can create another rule to reduce the number of Lambda function invocations if your rule handles multiple data points that have the same timestamp in each payload.
To do so, create a rule with a republish action that invokes the Lambda and publishes
the original payload with the timestamp string converted to timeInSeconds
and offsetInNanos
. Then, create a rule with an Amazon IoT SiteWise rule action to
consume the converted payload. With this approach, you reduce the number of times that
the rule invokes the Lambda but increase the number of Amazon IoT rule actions run. Consider
the pricing of each service if you apply this solution to your use case.
Example rule configurations
This section contains example rule configurations that you can use to create a rule with an Amazon IoT SiteWise action.
Example rule action that uses property aliases as message topics
The following example creates a rule with an Amazon IoT SiteWise action that uses the topic (through topic()) as the property alias to identify asset properties. You can use this example to define one rule for ingesting double-type data to all wind turbines in all wind farms. This example requires that you define property aliases on all turbine assets' properties. You would need to define a second, similar rule to ingest integer-type data.
aws iot create-topic-rule \ --rule-name SiteWiseWindFarmRule \ --topic-rule-payload file://sitewise-rule-payload.json
The example payload in sitewise-rule-payload.json
contains the
following content.
{ "sql": "SELECT * FROM '/company/windfarm/+/turbine/+/+' WHERE type = 'double'", "description": "Sends data to the wind turbine asset property with the same alias as the topic", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [ { "iotSiteWise": { "putAssetPropertyValueEntries": [ { "propertyAlias": "${topic()}", "propertyValues": [ { "timestamp": { "timeInSeconds": "${timeInSeconds}" }, "value": { "doubleValue": "${value}" } } ] } ], "roleArn": "arn:aws:iam::
account-id
:role/MySiteWiseActionRole
" } } ] }
With this rule action, you can send the following message to a wind turbine property
alias (for example, /company/windfarm/3/turbine/7/temperature
) as a topic to ingest data.
{ "type": "double", "value": "38.3", "timeInSeconds": "1581368533" }
Example rule action that uses timestamp() to determine time
The following example creates a rule with an Amazon IoT SiteWise action that identifies an asset property by IDs and uses timestamp() to determine the current time.
aws iot create-topic-rule \ --rule-name SiteWiseAssetPropertyRule \ --topic-rule-payload file://sitewise-rule-payload.json
The example payload in sitewise-rule-payload.json
contains the
following content.
{ "sql": "SELECT * FROM 'my/asset/property/topic'", "description": "Sends device data to an asset property", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [ { "iotSiteWise": { "putAssetPropertyValueEntries": [ { "assetId": "a1b2c3d4-5678-90ab-cdef-22222EXAMPLE", "propertyId": "a1b2c3d4-5678-90ab-cdef-33333EXAMPLE", "propertyValues": [ { "timestamp": { "timeInSeconds": "${floor(timestamp() / 1E3)}", "offsetInNanos": "${(timestamp() % 1E3) * 1E6}" }, "value": { "doubleValue": "${value}" } } ] } ], "roleArn": "arn:aws:iam::
account-id
:role/MySiteWiseActionRole
" } } ] }
With this rule action, you can send the following message to the
my/asset/property/topic
to ingest data.
{ "type": "double", "value": "38.3" }
Reducing costs with basic ingest
Amazon IoT Core provides a feature called Basic Ingest that you can use to send data through
Amazon IoT Core without incurring Amazon IoT messaging
costs
To use Basic Ingest, you send messages directly to a specific rule using a special
topic, $aws/rules/
. For example, to send a
message to a rule named rule-name
SiteWiseWindFarmRule
, you send a message to the topic
$aws/rules/SiteWiseWindFarmRule
.
If your rule action uses substitution templates that contain topic(Decimal), you can pass the original topic at the end of the Basic Ingest
special topic, such as
$aws/rules/
.
For example, to use Basic Ingest with the wind farm property alias example from the previous
section, you can send messages to the following topic.rule-name
/original-topic
$aws/rules/SiteWiseWindFarmRule//company/windfarm/3/turbine/7/temperature
Note
The above example includes a second slash (//
) because Amazon IoT removes the
Basic Ingest prefix ($aws/rules/
) from
the topic that's visible to the rule action. In this example, the rule receives the topic
rule-name
//company/windfarm/3/turbine/7/temperature
.
For more information, see Reducing messaging costs with basic ingest in the Amazon IoT Developer Guide.
Troubleshooting the Amazon IoT SiteWise rule action
To troubleshoot your Amazon IoT SiteWise rule action in Amazon IoT Core, you can configure CloudWatch Logs or you can configure a republish error action for your rule. For more information, see Troubleshooting an Amazon IoT SiteWise rule action.