Ingest data to Amazon IoT SiteWise from Amazon IoT things

Sign in to the Amazon Web Services Management Console.

Review the Amazon Regions where Amazon IoT SiteWise is supported. Switch to one of these supported Regions, if necessary.

Navigate to the Amazon IoT console. If a Connect device button appears, choose it.

In the left navigation pane, choose Security and then choose Policies.

Choose Create.

Enter a name for the Amazon IoT policy (for example, SiteWiseTutorialDevicePolicy).

Under Policy document, choose JSON to enter the following policy in JSON form. Replace region and account-id with your Region and account ID, such as us-east-1 and 123456789012.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "iot:Connect",
      "Resource": "arn:aws:iot:region:account-id:client/SiteWiseTutorialDevice*"
    },
    {
      "Effect": "Allow",
      "Action": "iot:Publish",
      "Resource": [
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/delete",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/get"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Receive",
      "Resource": [
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update/accepted",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/delete/accepted",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/get/accepted",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update/rejected",
        "arn:aws:iot:region:account-id:topic/$aws/things/${iot:Connection.Thing.ThingName}/shadow/delete/rejected"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iot:Subscribe",
      "Resource": [
        "arn:aws:iot:region:account-id:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update/accepted",
        "arn:aws:iot:region:account-id:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/delete/accepted",
        "arn:aws:iot:region:account-id:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/get/accepted",
        "arn:aws:iot:region:account-id:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/update/rejected",
        "arn:aws:iot:region:account-id:topicfilter/$aws/things/${iot:Connection.Thing.ThingName}/shadow/delete/rejected"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iot:GetThingShadow",
        "iot:UpdateThingShadow",
        "iot:DeleteThingShadow"
      ],
      "Resource": "arn:aws:iot:region:account-id:thing/SiteWiseTutorialDevice*"
    }
  ]
}

This policy enables your Amazon IoT devices to establish connections and communicate with device shadows using MQTT messages. For more information about MQTT messages, see What is MQTT?. To interact with device shadows, your Amazon IoT things publish and receive MQTT messages on topics that start with $aws/things/thing-name/shadow/. This policy incorporates a thing policy variable known as ${iot:Connection.Thing.ThingName}. This variable substitutes the connected thing's name in each topic. The iot:Connect statement sets limitations on which devices can establish connections, ensuring that the thing policy variable can only substitute names starting with SiteWiseTutorialDevice.

For more information, see Thing policy variables in the Amazon IoT Developer Guide.

Choose Create.

Open a command line and run the following command to create a directory for this tutorial.

mkdir iot-sitewise-rule-tutorial
cd iot-sitewise-rule-tutorial

Run the following command to create a directory for your thing's certificates.

mkdir device1

If you're creating additional things, increment the number in the directory name accordingly to keep track of which certificates belong to which thing.

Navigate to the Amazon IoT console.

In the left navigation pane, choose All devices in the Manage section. Then choose Things.

If a You don't have any things yet dialog box appears, choose Create a thing. Otherwise, choose Create things.

On the Creating things page, choose Create a single thing and then choose Next.

On the Specify thing properties page, enter a name for your Amazon IoT thing (for example, SiteWiseTutorialDevice1) and then choose Next. If you're creating additional things, increment the number in the thing name accordingly.

On the Configure device certificate - optional page, choose Auto-generate a new certificate (recommended), then choose Next. Certificates enable Amazon IoT to securely identify your devices.

On the Attach policies to certificate - optional page, select the policy you created in Step 1: Creating an Amazon IoT policy and choose Create thing.

On the Download certificates and keys dialog box, do the following:

1. Choose the Download links to download your thing's certificate, public key, and private key. Save all three files to the directory that you created for your thing's certificates (for example, iot-sitewise-rule-tutorial/device1).

   Important

   This is the only time that you can download your thing's certificate and keys, which you need for your device to successfully connect to Amazon IoT.

2. Choose the Download link to download a root CA certificate. Save the root CA certificate to the iot-sitewise-rule-tutorial. We recommend downloading Amazon Root CA 1.

Choose Done.

Navigate to the Amazon IoT SiteWise console.

In the left navigation pane, choose Models.

Choose Create model.

Under Model details, enter a name for your model. For example, SiteWise Tutorial Device Model.

Under Measurement definitions, do the following:

1. In Name, enter CPU Usage.

2. In Unit, enter %.

3. Leave the Data type as Double.

Measurement properties represent a device's raw data streams. For more information, see Define data streams from equipment (measurements).

Choose Add new measurement to add a second measurement property.

In the second row under Measurement definitions, do the following:

1. In Name, enter Memory Usage.

2. In Unit, enter %.

3. Leave the Data type as Double.

Under Metric definitions, do the following:

1. In Name, enter Average CPU Usage.

2. In Formula, enter avg(CPU Usage). Choose CPU Usage from the autocomplete list when it appears.

3. In Time interval, enter 5 minutes.

Metric properties define aggregation calculations that process all input data points over an interval and output a single data point per interval. This metric property calculates each device's average CPU usage every 5 minutes. For more information, see Aggregate data from properties and other assets (metrics).

Choose Add new metric to add a second metric property.

In the second row under Metric definitions, do the following:

1. In Name, enter Average Memory Usage.

2. In Formula, enter avg(Memory Usage). Choose Memory Usage from the autocomplete list when it appears.

3. In Time interval, enter 5 minutes.

This metric property calculates each device's average memory usage every 5 minutes.

(Optional) Add other additional metrics that you're interested in calculating per device. Some interesting functions include min and max. For more information, see Use formula expressions. In Step 4: Creating a device fleet asset model, you create a parent asset that can calculate metrics using data from your entire fleet of devices.

Choose Create model.

Navigate to the Amazon IoT SiteWise console.

In the left navigation pane, choose Models.

Choose Create model.

Under Model details, enter a name for your model. For example, SiteWise Tutorial Device Fleet Model.

Under Hierarchy definitions, do the following:

1. In Hierarchy name, enter Device.

2. In Hierarchy model, choose your device asset model (SiteWise Tutorial Device Model).

A hierarchy defines a relationship between a parent (fleet) asset model and a child (device) asset model. Parent assets can access child assets' property data. When you create assets later, you need to associate child assets to parent assets according to a hierarchy definition in the parent asset model. For more information, see Define asset model hierarchies.

Under Metric definitions, do the following:

1. In Name, enter Average CPU Usage.

2. In Formula, enter avg(Device | Average CPU Usage). When the autocomplete list appears, choose Device to choose a hierarchy, then choose Average CPU Usage to choose the metric from the device asset that you created earlier.

3. In Time interval, enter 5 minutes.

This metric property calculates the average CPU usage of all device assets associated to a fleet asset through the Device hierarchy.

Choose Add new metric to add a second metric property.

In the second row under Metric definitions, do the following:

1. In Name, enter Average Memory Usage.

2. In Formula, enter avg(Device | Average Memory Usage). When the autocomplete list appears, choose Device to choose a hierarchy, then choose Average Memory Usage to choose the metric from the device asset that you created earlier.

3. In Time interval, enter 5 minutes.

This metric property calculates the average memory usage of all device assets associated to a fleet asset through the Device hierarchy.

(Optional) Add other additional metrics that you're interested in calculating across your fleet of devices.

Choose Create model.

Navigate to the Amazon IoT SiteWise console.

In the left navigation pane, choose Assets.

Choose Create asset.

Under Model information, choose your device asset model, SiteWise Tutorial Device Model.

Under Asset information, enter a name for your asset. For example, SiteWise Tutorial Device 1.

Choose Create asset.

For your new device asset, choose Edit.

Under CPU Usage, enter /tutorial/device/SiteWiseTutorialDevice1/cpu as the property alias. You include the Amazon IoT thing's name in the property alias, so that you can ingest data from all of your devices using a single Amazon IoT rule.

Under Memory Usage, enter /tutorial/device/SiteWiseTutorialDevice1/memory as the property alias.

Choose Save.

Navigate to the Amazon IoT SiteWise console.

In the left navigation pane, choose Assets.

Choose Create asset.

Under Model information, choose your device fleet asset model, SiteWise Tutorial Device Fleet Model.

Under Asset information, enter a name for your asset. For example, SiteWise Tutorial Device Fleet 1.

Choose Create asset.

For your new device fleet asset, choose Edit.

Under Assets associated to this asset, choose Add associated asset and do the following:

1. Under Hierarchy, choose Device. This hierarchy identifies the hierarchical relationship between device and device fleet assets. You defined this hierarchy in the device fleet asset model earlier in this tutorial.

2. Under Asset, choose your device asset, SiteWise Tutorial Device 1.

(Optional) If you created multiple device assets earlier, repeat steps 8 through 10 for each device asset that you created.

Choose Save.

You should now see your device assets organized as a hierarchy.

Navigate to the Amazon IoT console.

In the left navigation pane, choose Message routing and then choose Rules.

Choose Create rule.

Enter a name and description for your rule and the choose Next.

Enter the following SQL statement and the choose Next.

SELECT
  *
FROM
  '$aws/things/+/shadow/update/accepted'
WHERE
  startsWith(topic(3), "SiteWiseTutorialDevice")

This rule query statement works because the device shadow service publishes shadow updates to $aws/things/thingName/shadow/update/accepted. For more information about device shadows, see Device shadow service in the Amazon IoT Developer Guide.

In the WHERE clause, this rule query statement uses the topic(3) function to get the thing name from the third segment of the topic. Then, the statement filters out devices that have names that don't match those of the tutorial devices. For more information about Amazon IoT SQL, see Amazon IoT SQL reference in the Amazon IoT Developer Guide.

Under Rule actions, choose Send message data to asset properties in Amazon IoT SiteWise and do the following:

1. Choose By property alias.

2. In Property alias, enter /tutorial/device/${topic(3)}/cpu.

   The ${...} syntax is a substitution template. Amazon IoT evaluates the contents within the braces. This substitution template pulls the thing name from the topic to create an alias unique to each thing. For more information, see Substitution templates in the Amazon IoT Developer Guide.

   Note

   Because an expression in a substitution template is evaluated separately from the SELECT statement, you can't use a substitution template to reference an alias created using an AS clause. You can reference only information present in the original payload, in addition to supported functions and operators.

3. In Entry ID - optional, enter ${concat(topic(3), "-cpu-", floor(state.reported.timestamp))}.

   Entry IDs uniquely identify each value entry attempt. If an entry returns an error, you can find the entry ID in the error output to troubleshoot the issue. The substitution template in this entry ID combines the thing name and the device's reported timestamp. For example, the resulting entry ID might look like SiteWiseTutorialDevice1-cpu-1579808494.

4. In Time in seconds, enter ${floor(state.reported.timestamp)}.

   This substitution template calculates the time in seconds from the device's reported timestamp. In this tutorial, devices report timestamp in seconds in Unix epoch time as a floating point number.

5. In Offset in nanos - optional, enter ${floor((state.reported.timestamp % 1) * 1E9)}.

   This substitution template calculates the nanosecond offset from the time in seconds by converting the decimal portion of the device's reported timestamp.

   Note

   Amazon IoT SiteWise requires that your data has a current timestamp in Unix epoch time. If your devices don't report time accurately, you can get the current time from the Amazon IoT rules engine with timestamp(). This function reports time in milliseconds, so you must update your rule action's time parameters to the following values:

   • In Time in seconds, enter ${floor(timestamp() / 1E3)}.

   • In Offset in nanos, enter ${(timestamp() % 1E3) * 1E6}.

6. In Data type, choose Double.

   This data type must match the data type of the asset property you defined in the asset model.

7. In Value, enter ${state.reported.cpu}. In substitution templates, you use the . operator to retrieve a value from within a JSON structure.

8. Choose Add entry to add a new entry for the memory usage property, and complete the following steps again for that property:

   1. Choose By property alias.

   2. In Property alias, enter /tutorial/device/${topic(3)}/memory.

   3. In Entry ID - optional, enter ${concat(topic(3), "-memory-", floor(state.reported.timestamp))}.

   4. In Time in seconds, enter ${floor(state.reported.timestamp)}.

   5. In Offset in nanos - optional, enter ${floor((state.reported.timestamp % 1) * 1E9)}.

   6. In Data type, choose Double.

   7. In Value, enter ${state.reported.memory}.

9. Under IAM Role, choose Create new role to create an IAM role for this rule action. This role allows Amazon IoT to push data to properties in your device fleet asset and its asset hierarchy.

10. Enter a role name and choose Create.

(Optional) Configure an error action that you can use to troubleshoot your rule. For more information, see Troubleshoot a rule (Amazon IoT SiteWise).

Choose Next.

Review the settings and choose Create to create the rule.

Navigate to the Amazon IoT console.

At the bottom of the left navigation pane, choose Settings.

Save your custom endpoint for use with the device client script. You use this endpoint to interact with your thing's shadows. This endpoint is unique to your account in the current Region.

Your custom endpoint should look like the following example.

identifier.iot.region.amazonaws.com

Open a command line and run the following command to navigate to the tutorial directory you created earlier.

cd iot-sitewise-rule-tutorial

Run the following command to install the Amazon IoT Device SDK for Python.

pip3 install AWSIoTPythonSDK

For more information, see Amazon IoT Device SDK for Python in the Amazon IoT Developer Guide

Run the following command to install psutil, a cross-platform process and system utilities library.

pip3 install psutil

For more information, see psutil in the Python Package Index.

Create a file called thing_performance.py in the iot-sitewise-rule-tutorial directory and then copy the following Python code into the file.

import AWSIoTPythonSDK.MQTTLib as AWSIoTPyMQTT

import json
import psutil
import argparse
import logging
import time


# Configures the argument parser for this program.
def configureParser():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-e",
        "--endpoint",
        action="store",
        required=True,
        dest="host",
        help="Your AWS IoT custom endpoint",
    )
    parser.add_argument(
        "-r",
        "--rootCA",
        action="store",
        required=True,
        dest="rootCAPath",
        help="Root CA file path",
    )
    parser.add_argument(
        "-c",
        "--cert",
        action="store",
        required=True,
        dest="certificatePath",
        help="Certificate file path",
    )
    parser.add_argument(
        "-k",
        "--key",
        action="store",
        required=True,
        dest="privateKeyPath",
        help="Private key file path",
    )
    parser.add_argument(
        "-p",
        "--port",
        action="store",
        dest="port",
        type=int,
        default=8883,
        help="Port number override",
    )
    parser.add_argument(
        "-n",
        "--thingName",
        action="store",
        required=True,
        dest="thingName",
        help="Targeted thing name",
    )
    parser.add_argument(
        "-d",
        "--requestDelay",
        action="store",
        dest="requestDelay",
        type=float,
        default=1,
        help="Time between requests (in seconds)",
    )
    parser.add_argument(
        "-v",
        "--enableLogging",
        action="store_true",
        dest="enableLogging",
        help="Enable logging for the AWS IoT Device SDK for Python",
    )
    return parser


# An MQTT shadow client that uploads device performance data to AWS IoT at a regular interval.
class PerformanceShadowClient:
    def __init__(
        self,
        thingName,
        host,
        port,
        rootCAPath,
        privateKeyPath,
        certificatePath,
        requestDelay,
    ):
        self.thingName = thingName
        self.host = host
        self.port = port
        self.rootCAPath = rootCAPath
        self.privateKeyPath = privateKeyPath
        self.certificatePath = certificatePath
        self.requestDelay = requestDelay

    # Updates this thing's shadow with system performance data at a regular interval.
    def run(self):
        print("Connecting MQTT client for {}...".format(self.thingName))
        mqttClient = self.configureMQTTClient()
        mqttClient.connect()
        print("MQTT client for {} connected".format(self.thingName))
        deviceShadowHandler = mqttClient.createShadowHandlerWithName(
            self.thingName, True
        )

        print("Running performance shadow client for {}...\n".format(self.thingName))
        while True:
            performance = self.readPerformance()
            print("[{}]".format(self.thingName))
            print("CPU:\t{}%".format(performance["cpu"]))
            print("Memory:\t{}%\n".format(performance["memory"]))
            payload = {"state": {"reported": performance}}
            deviceShadowHandler.shadowUpdate(
                json.dumps(payload), self.shadowUpdateCallback, 5
            )
            time.sleep(args.requestDelay)

    # Configures the MQTT shadow client for this thing.
    def configureMQTTClient(self):
        mqttClient = AWSIoTPyMQTT.AWSIoTMQTTShadowClient(self.thingName)
        mqttClient.configureEndpoint(self.host, self.port)
        mqttClient.configureCredentials(
            self.rootCAPath, self.privateKeyPath, self.certificatePath
        )
        mqttClient.configureAutoReconnectBackoffTime(1, 32, 20)
        mqttClient.configureConnectDisconnectTimeout(10)
        mqttClient.configureMQTTOperationTimeout(5)
        return mqttClient

    # Returns the local device's CPU usage, memory usage, and timestamp.
    def readPerformance(self):
        cpu = psutil.cpu_percent()
        memory = psutil.virtual_memory().percent
        timestamp = time.time()
        return {"cpu": cpu, "memory": memory, "timestamp": timestamp}

    # Prints the result of a shadow update call.
    def shadowUpdateCallback(self, payload, responseStatus, token):
        print("[{}]".format(self.thingName))
        print("Update request {} {}\n".format(token, responseStatus))


# Configures debug logging for the AWS IoT Device SDK for Python.
def configureLogging():
    logger = logging.getLogger("AWSIoTPythonSDK.core")
    logger.setLevel(logging.DEBUG)
    streamHandler = logging.StreamHandler()
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
    )
    streamHandler.setFormatter(formatter)
    logger.addHandler(streamHandler)


# Runs the performance shadow client with user arguments.
if __name__ == "__main__":
    parser = configureParser()
    args = parser.parse_args()
    if args.enableLogging:
        configureLogging()
    thingClient = PerformanceShadowClient(
        args.thingName,
        args.host,
        args.port,
        args.rootCAPath,
        args.privateKeyPath,
        args.certificatePath,
        args.requestDelay,
    )
    thingClient.run()

Run thing_performance.py from the command line with the following parameters:

Your command should look similar to the following example.

python3 thing_performance.py \
  --thingName SiteWiseTutorialDevice1 \
  --endpoint identifier.iot.region.amazonaws.com \
  --rootCA AmazonRootCA1.pem \
  --cert device1/thing-id-certificate.pem.crt \
  --key device1/thing-id-private.pem.key

If you're running the script for additional Amazon IoT things, update the thing name and certificate directory accordingly.

Try opening and closing programs on your device to see how the CPU and memory usages change. The script prints each CPU and memory usage reading. If the script uploads data to the device shadow service successfully, the script's output should look like the following example.

[SiteWiseTutorialDevice1]
CPU:    24.6%
Memory: 85.2%

[SiteWiseTutorialDevice1]
Update request e6686e44-fca0-44db-aa48-3ca81726f3e3 accepted

Follow these steps to verify that the script is updating the device shadow:

1. Navigate to the Amazon IoT console.

2. In the left navigation pane, choose All devices and then choose Things.

3. Choose your thing, SiteWiseTutorialDevice.

4. Choose the Device Shadows tab, choose Classic Shadow, and verify that the Shadow state looks like the following example.

   {
     "reported": {
       "cpu": 24.6,
       "memory": 85.2,
       "timestamp": 1579567542.2835066
     }
   }

   If your thing's shadow state is empty or doesn't look like the previous example, check that the script is running and successfully connected to Amazon IoT. If the script continues to time out when connecting to Amazon IoT, check that your thing policy is configured according to this tutorial.

Follow these steps to verify that the rule action is sending data to Amazon IoT SiteWise:

1. Navigate to the Amazon IoT SiteWise console.

2. In the left navigation pane, choose Assets.

3. Choose the arrow next to your device fleet asset (SiteWise Tutorial Device Fleet 1 1) to expand its asset hierarchy, and then choose your device asset (SiteWise Tutorial Device 1).

4. Choose Measurements.

5. Verify that the Latest value cells have values for the CPU Usage and Memory Usage properties.

   

6. If the CPU Usage and Memory Usage properties don't have the latest values, refresh the page. If values don't appear after a few minutes, see Troubleshoot a rule (Amazon IoT SiteWise).

Navigate to the Amazon IoT SiteWise console

In the left navigation pane, choose Assets.

When you delete assets in Amazon IoT SiteWise, you must first disassociate them.

Complete the following steps to disassociate your device assets from your device fleet asset:

1. Choose your device fleet asset ( SiteWise Tutorial Device Fleet 1).

2. Choose Edit.

3. Under Assets associated to this asset, choose Disassociate for each device asset associated to this device fleet asset.

4. Choose Save.

   You should now see your device assets no longer organized as a hierarchy.

Choose your device asset (SiteWise Tutorial Device 1).

Choose Delete.

In the confirmation dialog, enter Delete and then choose Delete.

Repeat steps 4 through 6 for each device asset and the device fleet asset (SiteWise Tutorial Device Fleet 1).

Navigate to the Amazon IoT SiteWise console.

If you haven't already, delete your device and device fleet assets. For more information, see the previous procedure. You can't delete a model if you have assets that were created from that model.

In the left navigation pane, choose Models.

Choose your device fleet asset model (SiteWise Tutorial Device Fleet Model).

When deleting hierarchical asset models, start by deleting the parent asset model first.

Choose Delete.

In the confirmation dialog, enter Delete and then choose Delete.

Repeat steps 4 through 6 for your device asset model (SiteWise Tutorial Device Model).

Navigate to the Amazon IoT console.

In the left navigation pane, choose Message routing and then choose Rules.

Select your rule and choose Delete.

In the confirmation dialog, enter the name of the rule and then choose Delete.