# Retaining device state while the device is offline with Device Shadows These tutorials show you how to use the Amazon IoT Device Shadow service to store and update the state information of a device. The Shadow document, which is a JSON document, shows the change in the device's state based on the messages published by a device, local app, or service. In this tutorial, the Shadow document shows the change in the color of a light bulb. These tutorials also show how the shadow stores this information even when the device is disconnected from the internet, and passes the latest state information back to the device when it comes back online and requests this information. We recommend that you try these tutorials in the order they're shown here, starting with the Amazon IoT resources you need to create and the necessary hardware setup, which also helps you learn the concepts incrementally. These tutorials show how to configure and connect a Raspberry Pi device for use with Amazon IoT. If you don't have the required hardware, you can follow these tutorials by adapting them to a device of your choice or by [creating a virtual device with Amazon EC2](creating-a-virtual-thing.md). **Tutorial scenario overview** The scenario for these tutorials is a local app or service that changes the color of a light bulb and that publishes its data to reserved shadow topics. These tutorials are similar to the Device Shadow functionality described in the [interactive getting started tutorial](interactive-demo.md) and are implemented on a Raspberry Pi device. The tutorials in this section focus on a single, classic shadow while showing how you might accommodate named shadows or multiple devices. The following tutorials will help you learn how to use the Amazon IoT Device Shadow service. + **[Tutorial: Preparing your Raspberry Pi to run the shadow application](create-resources-shadow.md)** This tutorial shows how to set up a Raspberry Pi device for connecting with Amazon IoT. You'll also create an Amazon IoT policy document and a thing resource, download the certificates, and then attach the policy to that thing resource. This tutorial takes about 30 minutes to complete. + **[Tutorial: Installing the Device SDK and running the sample application for Device Shadows](lightbulb-shadow-application.md)** This tutorial shows how to install the required tools, software, and the Amazon IoT Device SDK for Python, and then run the sample shadow application. This tutorial builds on concepts presented in [Connect a Raspberry Pi or other device](connecting-to-existing-device.md) and takes 20 minutes to complete. + **[Tutorial: Interacting with Device Shadow using the sample app and the MQTT test client](interact-lights-device-shadows.md)** This tutorial shows how you use the `shadow.py` sample app and **Amazon IoT console** to observe the interaction between Amazon IoT Device Shadows and the state changes of the light bulb. The tutorial also shows how to send MQTT messages to the Device Shadow's reserved topics. This tutorial can take 45 minutes to complete. **Amazon IoT Device Shadow overview** A Device Shadow is a persistent, virtual representation of a device that is managed by a [thing resource](iot-thing-management.md) you create in the Amazon IoT registry. The Shadow document is a JSON or a JavaScript notation doc that is used to store and retrieve the current state information for a device. You can use the shadow to get and set the state of a device over MQTT topics or HTTP REST APIs, regardless of whether the device is connected to the internet. A Shadow document contains a `state` property that describes these aspects of the device's state. + `desired`: Apps specify the desired states of device properties by updating the `desired` object. + `reported`: Devices report their current state in the `reported` object. + `delta`: Amazon IoT reports differences between the desired and the reported state in the `delta` object. Here is an example of a Shadow state document. ``` { "state": { "desired": { "color": "green" }, "reported": { "color": "blue" }, "delta": { "color": "green" } } } ``` To update a device's Shadow document, you can use the [reserved MQTT topics](reserved-topics.md#reserved-topics-shadow), the [Device Shadow REST APIs](device-shadow-rest-api.md) that support the `GET`, `UPDATE`, and `DELETE` operations with HTTP, and the [Amazon IoT CLI](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/iot-data/index.html). In the previous example, say you want to change the `desired` color to `yellow`. To do this, send a request to the [UpdateThingShadow](device-shadow-rest-api.md#API_UpdateThingShadow) API or publish a message to the [Update](device-shadow-mqtt.md#update-pub-sub-topic) topic, `$aws/things/THING_NAME/shadow/update`. ``` { "state": { "desired": { "color": yellow } } } ``` Updates affect only the fields specified in the request. After successfully updating the Device Shadow, Amazon IoT publishes the new `desired` state to the `delta` topic, `$aws/things/THING_NAME/shadow/delta`. The Shadow document in this case looks like this: ``` { "state": { "desired": { "color": yellow }, "reported": { "color": green }, "delta": { "color": yellow } } } ``` The new state is then reported to the Amazon IoT Device Shadow using the `Update` topic `$aws/things/THING_NAME/shadow/update` with the following JSON message: ``` { "state": { "reported": { "color": yellow } } } ``` If you want to get the current state information, send a request to the [GetThingShadow](device-shadow-rest-api.md#API_GetThingShadow) API or publish an MQTT message to the [Get](device-shadow-mqtt.md#get-pub-sub-topic) topic, `$aws/things/THING_NAME/shadow/get`. For more information about using the Device Shadow service, see [Amazon IoT Device Shadow service](iot-device-shadows.md). For more information about using Device Shadows in devices, apps, and services, see [Using shadows in devices](device-shadow-comms-device.md) and [Using shadows in apps and services](device-shadow-comms-app.md). For information about interacting with Amazon IoT shadows, see [Interacting with shadows](device-shadow-data-flow.md). For information about the MQTT reserved topics and HTTP REST APIs, see [Device Shadow MQTT topics](device-shadow-mqtt.md) and [Device Shadow REST API](device-shadow-rest-api.md). # Tutorial: Preparing your Raspberry Pi to run the shadow application This tutorial demonstrates how to set up and configure a Raspberry Pi device and create the Amazon IoT resources that a device requires to connect and exchange MQTT messages. **Note** If you're planning to [Create a virtual device with Amazon EC2](creating-a-virtual-thing.md), you can skip this page and continue to [Configure your device](configure-device.md). You'll create these resources when you create your virtual thing. If you would like to use a different device instead of the Raspberry Pi, you can try to follow these tutorials by adapting them to a device of your choice. **In this tutorial, you'll learn how to:** + Set up a Raspberry Pi device and configure it for use with Amazon IoT. + Create an Amazon IoT policy document, which authorizes your device to interact with Amazon IoT services. + Create a thing resource in Amazon IoT the X.509 device certificates, and then attach the policy document. The thing is the virtual representation of your device in the Amazon IoT registry. The certificate authenticates your device to Amazon IoT Core, and the policy document authorizes your device to interact with Amazon IoT. **How to run this tutorial** To run the `shadow.py` sample application for Device Shadows, you'll need a Raspberry Pi device that connects to Amazon IoT. We recommend that you follow this tutorial in the order it's presented here, starting with setting up the Raspberry Pi and it's accessories, and then creating a policy and attaching the policy to a thing resource that you create. You can then follow this tutorial by using the graphical user interface (GUI) supported by the Raspberry Pi to open the Amazon IoT console on the device's web browser, which also makes it easier to download the certificates directly to your Raspberry Pi for connecting to Amazon IoT. **Before you start this tutorial, make sure that you have:** + An Amazon Web Services account. If you don't have one, complete the steps described in [Set up Amazon Web Services account](setting-up.md) before you continue. You'll need your Amazon Web Services account and Amazon IoT console to complete this tutorial. + The Raspberry Pi and its necessary accessories. You'll need: + A [Raspberry Pi 3 Model B](https://www.raspberrypi.com/products/) or more recent model. This tutorial might work on earlier versions of the Raspberry Pi, but we haven't tested it. + [Raspberry Pi OS (32-bit)](https://www.raspberrypi.com/software/operating-systems/) or later. We recommend using the latest version of the Raspberry Pi OS. Earlier versions of the OS might work, but we haven't tested it. + An Ethernet or Wi-Fi connection. + Keyboard, mouse, monitor, cables, and power supplies. This tutorial takes about 30 minutes to complete. ## Step 1: Set up and configure Raspberry Pi device In this section, we'll configure a Raspberry Pi device for use with Amazon IoT. **Important** Adapting these instructions to other devices and operating systems can be challenging. You'll need to understand your device well enough to be able to interpret these instructions and apply them to your device. If you encounter difficulties, you might try one of the other device options as an alternative, such as [Create a virtual device with Amazon EC2](creating-a-virtual-thing.md) or [Use your Windows or Linux PC or Mac as an Amazon IoT device](using-laptop-as-device.md). You'll need to configure your Raspberry Pi such that it can start the operating system (OS), connect to the internet, and allow you to interact with it at a command line interface. You can also use the graphical user interface (GUI) supported with the Raspberry Pi to open the Amazon IoT console and run the rest of this tutorial. **To set up the Raspberry Pi** 1. Insert the SD card into the MicroSD card slot on the Raspberry Pi. Some SD cards come pre-loaded with an installation manager that prompts you with a menu to install the OS after booting up the board. You can also use the Raspberry Pi imager to install the OS on your card. 1. Connect an HDMI TV or monitor to the HDMI cable that connects to the HDMI port of the Raspberry Pi. 1. Connect the keyboard and mouse to the USB ports of the Raspberry Pi and then plug in the power adapter to boot up the board. After the Raspberry Pi boots up, if the SD card came pre-loaded with the installation manager, a menu appears to install the operating system. If you have trouble installing the OS, you can try the following steps. For more information about setting up the Raspberry Pi, see [Setting up your Raspberry Pi](https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/). **If you're having trouble setting up the Raspberry Pi:** + Check whether you inserted the SD card before booting up the board. If you plug in the SD card after booting up the board, the installation menu might not appear. + Make sure that the TV or monitor is turned on and the correct input is selected. + Ensure that you are using Raspberry Pi compatible software. After you have installed and configured the Raspberry Pi OS, open the Raspberry Pi's web browser and navigate to the Amazon IoT Core console to continue the rest of the steps in this tutorial. If you can open the Amazon IoT Core console, you're Raspberry Pi is ready and you can continue to [Tutorial: Provisioning your device in Amazon IoT](shadow-provision-cloud.md). If you're having trouble or need additional help, see [Getting help for your Raspberry Pi](https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/5). # Tutorial: Provisioning your device in Amazon IoT This section creates the Amazon IoT Core resources that your tutorial will use. **Topics** + [ ## Step 1: Create an Amazon IoT policy for the Device Shadow ](#create-policy-shadow) + [ ## Step 2: Create a thing resource and attach the policy to the thing ](#create-thing-shadow) + [ ## Step 3: Review the results and next steps ](#resources-shadow-review) ## Step 1: Create an Amazon IoT policy for the Device Shadow X.509 certificates authenticate your device with Amazon IoT Core. Amazon IoT policies are attached to the certificate that permits the device to perform Amazon IoT operations, such as subscribing or publishing to MQTT reserved topics used by the Device Shadow service. Your device presents its certificate when it connects and sends messages to Amazon IoT Core. In this procedure, you'll create a policy that allows your device to perform the Amazon IoT operations necessary to run the example program. We recommend that you create a policy that grants only the permissions required to perform the task. You create the Amazon IoT policy first, and then attach it to the device certificate that you'll create later. **To create an Amazon IoT policy** 1. On the left menu, choose **Secure**, and then choose **Policies**. If your account has existing policies, choose **Create**, otherwise, on the **You don’t have a policy yet** page, choose **Create a policy**. 1. On the **Create a policy** page: 1. Enter a name for the policy in the **Name** field (for example, **My\$1Device\$1Shadow\$1policy**). Do not use personally identifiable information in your policy names. 1. In the policy document, you describe connect, subscribe, receive, and publish actions that give the device permission to publish and subscribe to the MQTT reserved topics. Copy the following sample policy and paste it in your policy document. Replace `thingname` with the name of the thing that you'll create (for example, `My_light_bulb`), `region` with the Amazon IoT Region where you're using the services, and `account` with your Amazon Web Services account number. For more information about Amazon IoT policies, see [Amazon IoT Core policies](iot-policies.md). **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iot:Publish" ], "Resource": [ "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/get", "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/update" ] }, { "Effect": "Allow", "Action": [ "iot:Receive" ], "Resource": [ "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/get/accepted", "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/get/rejected", "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/update/accepted", "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/update/rejected", "arn:aws-cn:iot:us-east-1:123456789012:topic/$aws/things/thingname/shadow/update/delta" ] }, { "Effect": "Allow", "Action": [ "iot:Subscribe" ], "Resource": [ "arn:aws-cn:iot:us-east-1:123456789012:topicfilter/$aws/things/thingname/shadow/get/accepted", "arn:aws-cn:iot:us-east-1:123456789012:topicfilter/$aws/things/thingname/shadow/get/rejected", "arn:aws-cn:iot:us-east-1:123456789012:topicfilter/$aws/things/thingname/shadow/update/accepted", "arn:aws-cn:iot:us-east-1:123456789012:topicfilter/$aws/things/thingname/shadow/update/rejected", "arn:aws-cn:iot:us-east-1:123456789012:topicfilter/$aws/things/thingname/shadow/update/delta" ] }, { "Effect": "Allow", "Action": "iot:Connect", "Resource": "arn:aws-cn:iot:us-east-1:123456789012:client/test-*" } ] } ``` ## Step 2: Create a thing resource and attach the policy to the thing Devices connected to Amazon IoT can be represented by *thing resources* in the Amazon IoT registry. A *thing resource* represents a specific device or logical entity, such as the light bulb in this tutorial. To learn how to create a thing in Amazon IoT, follow the steps described in [Create a thing object](create-iot-resources.md#create-aws-thing). Here are some key things to note as you follow the steps in that tutorial: 1. Choose **Create a single thing**, and in the **Name** field, enter a name for the thing that is the same as the `thingname` (for example, `My_light_bulb`) you specified when you created the policy earlier. You can't change a thing name after it has been created. If you gave it a different name other than `thingname`, create a new thing with name as `thingname` and delete the old thing. **Note** Do not use personally identifiable information in your thing name. The thing name can appear in unencrypted communications and reports. 1. We recommend that you download each of the certificate files on the **Certificate created\$1** page into a location where you can easily find them. You'll need to install these files for running the sample application. We recommend that you download the files into a `certs` subdirectory in your `home` directory on the Raspberry Pi and name each of them with a simpler name as suggested in the following table. **Certificate file names** [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/iot/latest/developerguide/shadow-provision-cloud.html) 1. After you activate the certificate to enable connections to Amazon IoT, choose **Attach a policy** and make sure you attach the policy that you created earlier (for example, **My\$1Device\$1Shadow\$1policy**) to the thing. After you've created a thing, you can see your thing resource displayed in the list of things in the Amazon IoT console. ## Step 3: Review the results and next steps **In this tutorial, you learned how to:** + Set up and configure the Raspberry Pi device. + Create an Amazon IoT policy document that authorizes your device to interact with Amazon IoT services. + Create a thing resource and associated X.509 device certificate, and attach the policy document to it. **Next steps** You can now install the Amazon IoT device SDK for Python, run the `shadow.py` sample application, and use Device Shadows to control the state. For more information about how to run this tutorial, see [Tutorial: Installing the Device SDK and running the sample application for Device Shadows](lightbulb-shadow-application.md). # Tutorial: Installing the Device SDK and running the sample application for Device Shadows This section shows how you can install the required software and the Amazon IoT Device SDK for Python and run the `shadow.py` sample application to edit the Shadow document and control the shadow's state. **In this tutorial, you'll learn how to:** + Use the installed software and Amazon IoT Device SDK for Python to run the sample app. + Learn how entering a value using the sample app publishes the desired value in the Amazon IoT console. + Review the `shadow.py` sample app and how it uses the MQTT protocol to update the shadow's state. **Before you run this tutorial:** You must have set up your Amazon Web Services account, configured your Raspberry Pi device, and created an Amazon IoT thing and policy that gives the device permissions to publish and subscribe to the MQTT reserved topics of the Device Shadow service. For more information, see [Tutorial: Preparing your Raspberry Pi to run the shadow application](create-resources-shadow.md). You must have also installed Git, Python, and the Amazon IoT Device SDK for Python. This tutorial builds on the concepts presented in the tutorial [Connect a Raspberry Pi or other device](connecting-to-existing-device.md). If you haven't tried that tutorial, we recommend that you follow the steps described in that tutorial to install the certificate files and Device SDK and then come back to this tutorial to run the `shadow.py` sample app. **Topics** + [ ## Step 1: Run the shadow.py sample app ](#run-sample-application-shadows) + [ ## Step 2: Review the shadow.py Device SDK sample app ](#review-shadow-sample-code) + [ ## Step 3: Troubleshoot problems with the `shadow.py` sample app ](#shadow-sample-app-troubleshoot) + [ ## Step 4: Review the results and next steps ](#sample-app-shadow-review) This tutorial takes about 20 minutes to complete. ## Step 1: Run the shadow.py sample app Before you run the `shadow.py` sample app, you'll need the following information in addition to the names and location of the certificate files that you installed. **Application parameter values** | Parameter | Where to find the value | | --- | --- | | your-iot-thing-name | Name of the Amazon IoT thing that you created earlier in [Step 2: Create a thing resource and attach the policy to the thing](shadow-provision-cloud.md#create-thing-shadow). To find this value, in the [Amazon IoT console](https://console.amazonaws.cn/iot/home), choose **Manage**, and then choose **Things**. | | your-iot-endpoint | The *your-iot-endpoint* value has a format of: `endpoint_id-ats.iot.region.amazonaws.com`, for example, `a3qj468EXAMPLE-ats.iot.us-west-2.amazonaws.com`. To find this value: [\[See the AWS documentation website for more details\]](http://docs.amazonaws.cn/en_us/iot/latest/developerguide/lightbulb-shadow-application.html) | **Install and run the sample app** 1. Navigate to the sample app directory. ``` cd ~/aws-iot-device-sdk-python-v2/samples/service-clients ``` 1. In the command line window, replace *your-iot-endpoint* and *your-iot-thing-name* as indicated and run this command. ``` python3 shadow.py --ca_file ~/certs/Amazon-root-CA-1.pem --cert ~/certs/device.pem.crt --key ~/certs/private.pem.key --endpoint your-iot-endpoint --thing_name your-iot-thing-name ``` 1. Observe that the sample app: 1. Connects to the Amazon IoT service for your account. 1. Subscribes to `Delta` events and `Update` and `Get` responses. 1. Prompts you to enter a desired value in the terminal. 1. Displays output similar to the following: ``` Connecting to a3qEXAMPLEffp-ats.iot.us-west-2.amazonaws.com with client ID 'test-0c8ae2ff-cc87-49d2-a82a-ae7ba1d0ca5a'... Connected! Subscribing to Delta events... Subscribing to Update responses... Subscribing to Get responses... Requesting current shadow state... Launching thread to read user input... Finished getting initial shadow state. Shadow contains reported value 'off'. Enter desired value: ``` **Note** If you're having trouble running the `shadow.py` sample app, review [Step 3: Troubleshoot problems with the `shadow.py` sample app](#shadow-sample-app-troubleshoot). To get additional information that might help you correct the problem, add the `--verbosity debug` parameter to the command line so the sample app displays detailed messages about what it’s doing. **Enter values and observe the updates in Shadow document** You can enter values in the terminal to specify the `desired` value, which also updates the `reported` value. Say you enter the color `yellow` in the terminal. The `reported` value is also updated to the color `yellow`. The following shows the messages displayed in the terminal: ``` Enter desired value: yellow Changed local shadow value to 'yellow'. Updating reported shadow value to 'yellow'... Update request published. Finished updating reported shadow value to 'yellow'. ``` When you publish this update request, Amazon IoT creates a default, classic shadow for the thing resource. You can observe the update request that you published to the `reported` and `desired` values in the Amazon IoT console by looking at the Shadow document for the thing resource that you created (for example, `My_light_bulb`). To see the update in the Shadow document: 1. In the Amazon IoT console, choose **Manage** and then choose **Things**. 1. In the list of things displayed, select the thing that you created, choose **Shadows**, and then choose **Classic Shadow**. The Shadow document should look similar to the following, showing the `reported` and `desired` values set to the color `yellow`. You see these values in the **Shadow state** section of the document. ``` { "desired": { "welcome": "aws-iot", "color": "yellow" }, "reported": { "welcome": "aws-iot", "color": "yellow" } } ``` You also see a **Metadata** section that contains the timestamp information and version number of the request. You can use the state document version to ensure you are updating the most recent version of a device's Shadow document. If you send another update request, the version number increments by 1. When you supply a version with an update request, the service rejects the request with an HTTP 409 conflict response code if the current version of the state document doesn't match the version supplied. ``` { "metadata": { "desired": { "welcome": { "timestamp": 1620156892 }, "color": { "timestamp": 1620156893 } }, "reported": { "welcome": { "timestamp": 1620156892 }, "color": { "timestamp": 1620156893 } } }, "version": 10 } ``` To learn more about the Shadow document and observe changes to the state information, proceed to the next tutorial [Tutorial: Interacting with Device Shadow using the sample app and the MQTT test client](interact-lights-device-shadows.md) as described in the [Step 4: Review the results and next steps](#sample-app-shadow-review) section of this tutorial. Optionally, you can also learn about the `shadow.py` sample code and how it uses the MQTT protocol in the following section. ## Step 2: Review the shadow.py Device SDK sample app This section reviews the `shadow.py` sample app from the **Amazon IoT Device SDK v2 for Python** used in this tutorial. Here, we'll review how it connects to Amazon IoT Core by using the MQTT and MQTT over WSS protocol. The [Amazon common runtime (Amazon-CRT)](https://github.com/awslabs/aws-crt-python#aws-crt-python) library provides the low-level communication protocol support and is included with the Amazon IoT Device SDK v2 for Python. While this tutorial uses MQTT and MQTT over WSS, Amazon IoT supports devices that publish HTTPS requests. For an example of a Python program that sends an HTTP message from a device, see the [HTTPS code example](http.md#codeexample) using Python’s `requests` library. For information about how you can make an informed decision about which protocol to use for your device communications, review the [Choosing an application protocol for your device communication](protocols.md#protocol-selection). **MQTT** The `shadow.py` sample calls `mtls_from_path` (shown here) in the [https://github.com/awslabs/aws-crt-python/blob/89207bcf1387177034e02fe29e8e469ca45e39b7/awscrt/awsiot_mqtt_connection_builder.py](https://github.com/awslabs/aws-crt-python/blob/89207bcf1387177034e02fe29e8e469ca45e39b7/awscrt/awsiot_mqtt_connection_builder.py) to establish a connection with Amazon IoT Core by using the MQTT protocol. `mtls_from_path` uses X.509 certificates and TLS v1.2 to authenticate the device. The Amazon-CRT library handles the lower-level details of that connection. ``` mqtt_connection = mqtt_connection_builder.mtls_from_path( endpoint=args.endpoint, cert_filepath=args.cert, pri_key_filepath=args.key, ca_filepath=args.ca_file, client_bootstrap=client_bootstrap, on_connection_interrupted=on_connection_interrupted, on_connection_resumed=on_connection_resumed, client_id=args.client_id, clean_session=False, keep_alive_secs=6 ) ``` + `endpoint` is your Amazon IoT endpoint that you passed in from the command line and `client_id` is the ID that uniquely identifies this device in the Amazon Web Services Region. + `cert_filepath`, `pri_key_filepath`, and `ca_filepath` are the paths to the device's certificate and private key files, and the root CA file. + `client_bootstrap` is the common runtime object that handles socket communication activities, and is instantiated prior to the call to `mqtt_connection_builder.mtls_from_path`. + `on_connection_interrupted` and `on_connection_resumed` are callback functions to call when the device’s connection is interrupted and resumed. + `clean_session` is whether to start a new, persistent session, or if one is present, reconnect to an existing one. `keep_alive_secs` is the keep alive value, in seconds, to send in the `CONNECT` request. A ping will automatically be sent at this interval. The server assumes that the connection is lost if it doesn't receive a ping after 1.5 times this value. The `shadow.py` sample also calls `websockets_with_default_aws_signing` in the [https://github.com/awslabs/aws-crt-python/blob/89207bcf1387177034e02fe29e8e469ca45e39b7/awscrt/awsiot_mqtt_connection_builder.py](https://github.com/awslabs/aws-crt-python/blob/89207bcf1387177034e02fe29e8e469ca45e39b7/awscrt/awsiot_mqtt_connection_builder.py) to establish a connection with Amazon IoT Core using MQTT protocol over WSS. MQTT over WSS also uses the same parameters as MQTT and takes these additional parameters: + `region` is the Amazon signing Region used by Signature V4 authentication, and `credentials_provider` is the Amazon credentials provided to use for authentication. The Region is passed from the command line, and the `credentials_provider` object is instantiated just prior to the call to `mqtt_connection_builder.websockets_with_default_aws_signing`. + `websocket_proxy_options` is the HTTP proxy options, if using a proxy host. In the `shadow.py` sample app, this value is instantiated just prior to the call to `mqtt_connection_builder.websockets_with_default_aws_signing`. **Subscribe to Shadow topics and events** The `shadow.py` sample attempts to establish a connection and waits to be fully connected. If it's not connected, commands are queued up. Once connected, the sample subscribes to delta events and update and get messages, and publishes messages with a Quality of Service (QoS) level of 1 (`mqtt.QoS.AT_LEAST_ONCE`). When a device subscribes to a message with QoS level 1, the message broker saves the messages that the device is subscribed to until they can be sent to the device. The message broker resends the messages until it receives a `PUBACK` response from the device. For more information about the MQTT protocol, see [Review the MQTT protocol](sdk-tutorials.md#sdk-tutorials-mqtt-review) and [MQTT](mqtt.md). For more information about how MQTT, MQTT over WSS, persistent sessions, and QoS levels that are used in this tutorial, see [Review the pubsub.py Device SDK sample app](sdk-tutorials.md#sdk-tutorials-explore-sample). ## Step 3: Troubleshoot problems with the `shadow.py` sample app When you run the `shadow.py` sample app, you should see some messages displayed in the terminal and a prompt to enter a `desired` value. If the program throws an error, then to debug the error, you can start by checking whether you ran the correct command for your system. In some cases, the error message might indicate connection issues and look similar to: `Host name was invalid for dns resolution` or `Connection was closed unexpectedly`. In such cases, here are some things you can check: + **Check the endpoint address in the command** Review the `endpoint` argument in the command you entered to run the sample app, (for example, `a3qEXAMPLEffp-ats.iot.us-west-2.amazonaws.com`) and check this value in the **Amazon IoT console**. To check whether you used the correct value: 1. In the **Amazon IoT console**, choose **Manage** and then choose **Things**. 1. Choose the thing you created for your sample app (for example, **My\$1light\$1bulb**) and then choose **Interact**. On the thing details page, your endpoint is displayed in the **HTTPS** section. You should also see a message that says: `This thing already appears to be connected.` + **Check certificate activation** Certificates authenticate your device with Amazon IoT Core. To check whether your certificate is active: 1. In the **Amazon IoT console**, choose **Manage** and then choose **Things**. 1. Choose the thing you created for your sample app (for example, **My\$1light\$1bulb**) and then choose **Security**. 1. Select the certificate and then, from the certificate's details page, choose Select the certificate and then, from the certificate's details page, choose **Actions**. If in the dropdown list **Activate** isn't available and you can only choose **Deactivate**, your certificate is active. If not, choose **Activate** and rerun the sample program. If the program still doesn't run, check the certificate file names in the `certs` folder. + **Check the policy attached to the thing resource** While certificates authenticate your device, Amazon IoT policies permit the device to perform Amazon IoT operations, such as subscribing or publishing to MQTT reserved topics. To check whether the correct policy is attached: 1. Find the certificate as described previously, and then choose **Policies**. 1. Choose the policy displayed and check whether it describes the `connect`, `subscribe`, `receive`, and `publish` actions that give the device permission to publish and subscribe to the MQTT reserved topics. For a sample policy, see [Step 1: Create an Amazon IoT policy for the Device Shadow](shadow-provision-cloud.md#create-policy-shadow). If you see error messages that indicate trouble connecting to Amazon IoT, it could be because of the permissions you're using for the policy. If that's the case, we recommend that you start with a policy that provides full access to Amazon IoT resources and then rerun the sample program. You can either edit the current policy, or choose the current policy, choose **Detach**, and then create another policy that provides full access and attach it to your thing resource. You can later restrict the policy to only the actions and policies you need to run the program. **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iot:*" ], "Resource": "*" } ] } ``` + **Check your Device SDK installation** If the program still doesn't run, you can reinstall the Device SDK to make sure that your SDK installation is complete and correct. ## Step 4: Review the results and next steps **In this tutorial, you learned how to:** + Install the required software, tools, and the Amazon IoT Device SDK for Python. + Understand how the sample app, `shadow.py`, uses the MQTT protocol for retrieving and updating the shadow's current state. + Run the sample app for Device Shadows and observe the update to the Shadow document in the Amazon IoT console. You also learned to troubleshoot any issues and fix errors when running the program. **Next steps** You can now run the `shadow.py` sample application and use Device Shadows to control the state. You can observe the updates to the Shadow document in the Amazon IoT Console and observe delta events that the sample app responds to. Using the MQTT test client, you can subscribe to the reserved shadow topics and observe messages received by the topics when running the sample program. For more information about how to run this tutorial, see [Tutorial: Interacting with Device Shadow using the sample app and the MQTT test client](interact-lights-device-shadows.md). # Tutorial: Interacting with Device Shadow using the sample app and the MQTT test client To interact with the `shadow.py` sample app, enter a value in the terminal for the `desired` value. For example, you can specify colors that resemble the traffic lights and Amazon IoT responds to the request and updates the reported values. **In this tutorial, you'll learn how to:** + Use the `shadow.py` sample app to specify desired states and update the shadow's current state. + Edit the Shadow document to observe delta events and how the `shadow.py` sample app responds to it. + Use the MQTT test client to subscribe to shadow topics and observe updates when you run the sample program. **Before you run this tutorial, you must have:** Set up your Amazon Web Services account, configured your Raspberry Pi device, and created an Amazon IoT thing and policy. You must have also installed the required software, Device SDK, certificate files, and run the sample program in the terminal. For more information, see the previous tutorials [Tutorial: Preparing your Raspberry Pi to run the shadow application](create-resources-shadow.md) and [Step 1: Run the shadow.py sample app](lightbulb-shadow-application.md#run-sample-application-shadows). You must complete these tutorials if you haven't already. **Topics** + [ ## Step 1: Update desired and reported values using `shadow.py` sample app ](#update-desired-shadow-sample) + [ ## Step 2: View messages from the `shadow.py` sample app in the MQTT test client ](#shadow-sample-view-msg) + [ ## Step 3: Troubleshoot errors with Device Shadow interactions ](#shadow-observe-messages-troubleshoot) + [ ## Step 4: Review the results and next steps ](#sample-shadow-review) This tutorial takes about 45 minutes to complete. ## Step 1: Update desired and reported values using `shadow.py` sample app In the previous tutorial [Step 1: Run the shadow.py sample app](lightbulb-shadow-application.md#run-sample-application-shadows), you learned how to observe a message published to the Shadow document in the Amazon IoT console when you enter a desired value as described in the section [Tutorial: Installing the Device SDK and running the sample application for Device Shadows](lightbulb-shadow-application.md). In the previous example, we set the desired color to `yellow`. After you enter each value, the terminal prompts you to enter another `desired` value. If you again enter the same value (`yellow`), the app recognizes this and prompts you to enter a new `desired` value. ``` Enter desired value: yellow Local value is already 'yellow'. Enter desired value: ``` Now, say that you enter the color `green`. Amazon IoT responds to the request and updates the `reported` value to `green`. This is how the update happens when the `desired` state is different from the `reported` state, causing a delta. **How the `shadow.py` sample app simulates Device Shadow interactions:** 1. Enter a `desired` value (say `yellow`) in the terminal to publish the desired state. 1. As the `desired` state is different from the `reported` state (say the color `green`), a delta occurs, and the app that is subscribed to the delta receives this message. 1. The app responds to the message and updates its state to the `desired` value, `yellow`. 1. The app then publishes an update message with the new reported value of the device's state, `yellow`. Following shows the messages displayed in the terminal that shows how the update request is published. ``` Enter desired value: green Changed local shadow value to 'green'. Updating reported shadow value to 'green'... Update request published. Finished updating reported shadow value to 'green'. ``` In the Amazon IoT console, the Shadow document reflects the updated value to `green` for both the `reported` and `desired` fields, and the version number is incremented by 1. For example, if the previous version number was displayed as 10, the current version number will display as 11. **Note** Deleting a shadow doesn't reset the version number to 0. You'll see that the shadow version is incremented by 1 when you publish an update request or create another shadow with the same name. **Edit the Shadow document to observe delta events** The `shadow.py` sample app is also subscribed to `delta` events, and responds when there is a change to the `desired` value. For example, you can change the `desired` value to the color `red`. To do this, in the Amazon IoT console, edit the Shadow document by clicking **Edit** and then set the `desired` value to `red` in the JSON, while keeping the `reported` value to `green`. Before you save the changes, keep the terminal on the Raspberry Pi open as you'll see messages displayed in the terminal when the change occurs. ``` { "desired": { "welcome": "aws-iot", "color": "red" }, "reported": { "welcome": "aws-iot", "color": "green" } } ``` After you save the new value, the `shadow.py` sample app responds to this change and displays messages in the terminal indicating the delta. You should then see the following messages appear below the prompt for entering the `desired` value. ``` Enter desired value: Received shadow delta event. Delta reports that desired value is 'red'. Changing local value... Changed local shadow value to 'red'. Updating reported shadow value to 'red'... Finished updating reported shadow value to 'red'. Enter desired value: Update request published. Finished updating reported shadow value to 'red'. ``` ## Step 2: View messages from the `shadow.py` sample app in the MQTT test client You can use the **MQTT test client** in the **Amazon IoT console** to monitor MQTT messages that are passed in your Amazon Web Services account. By subscribing to reserved MQTT topics used by the Device Shadow service, you can observe the messages received by the topics when running the sample app. If you haven't already used the MQTT test client, you can review [View MQTT messages with the Amazon IoT MQTT client](view-mqtt-messages.md). This helps you learn how to use the **MQTT test client** in the **Amazon IoT console** to view MQTT messages as they pass through the message broker. 1. **Open the MQTT test client** Open the [MQTT test client in the Amazon IoT console](https://console.amazonaws.cn//iot/home#/test) in a new window so that you can observe the messages received by the MQTT topics without losing the configuration of your MQTT test client. The MQTT test client doesn't retain any subscriptions or message logs if you leave it to go to another page in the console. For this section of the tutorial, you can have the Shadow document of your Amazon IoT thing and the MQTT test client open in separate windows to more easily observe the interaction with Device Shadows. 1. **Subscribe to the MQTT reserved Shadow topics** You can use the MQTT test client to enter the names of the Device Shadow's MQTT reserved topics and subscribe to them to receive updates when running the `shadow.py` sample app. To subscribe to the topics: 1. In the **MQTT test client** in the **Amazon IoT console**, choose **Subscribe to a topic**. 1. In the **Topic filter** section, enter: **\$1aws/things/*thingname*/shadow/update/\$1**. Here, `thingname` is the name of the thing resource that you created earlier (for example, `My_light_bulb`). 1. Keep the default values for the additional configuration settings, and then choose **Subscribe**. By using the **\$1** wildcard in the topic subscription, you can subscribe to multiple MQTT topics at the same time and observe all the messages that are exchanged between the device and its Shadow in a single window. For more information about the wildcard characters and their use, see [MQTT topics](topics.md). 1. **Run `shadow.py` sample program and observe messages** In your command line window of the Raspberry Pi, if you've disconnected the program, run the sample app again and watch the messages in the **MQTT test client** in the **Amazon IoT console**. 1. Run the following command to restart the sample program. Replace *your-iot-thing-name* and *your-iot-endpoint* with the names of the Amazon IoT thing that you created earlier (for example, `My_light_bulb`), and the endpoint to interact with the device. ``` cd ~/aws-iot-device-sdk-python-v2/samples/service-clients python3 shadow.py --ca_file ~/certs/Amazon-root-CA-1.pem --cert ~/certs/device.pem.crt --key ~/certs/private.pem.key --endpoint your-iot-endpoint --thing_name your-iot-thing-name ``` The `shadow.py` sample app then runs and retrieves the current shadow state. If you've deleted the shadow or cleared the current states, the program sets the current value to `off` and then prompts you to enter a `desired` value. ``` Connecting to a3qEXAMPLEffp-ats.iot.us-west-2.amazonaws.com with client ID 'test-0c8ae2ff-cc87-49d2-a82a-ae7ba1d0ca5a'... Connected! Subscribing to Delta events... Subscribing to Update responses... Subscribing to Get responses... Requesting current shadow state... Launching thread to read user input... Finished getting initial shadow state. Shadow document lacks 'color' property. Setting defaults... Changed local shadow value to 'off'. Updating reported shadow value to 'off'... Update request published. Finished updating reported shadow value to 'off'... Enter desired value: ``` On the other hand, if the program was running and you restarted it, you'll see the latest color value reported in the terminal. In the MQTT test client, you'll see an update to the topics **\$1aws/things/*thingname*/shadow/get** and **\$1aws/things/*thingname*/shadow/get/accepted**. Suppose that the latest color reported was `green`. Following shows the contents of the **\$1aws/things/*thingname*/shadow/get/accepted** JSON file. ``` { "state": { "desired": { "welcome": "aws-iot", "color": "green" }, "reported": { "welcome": "aws-iot", "color": "green" } }, "metadata": { "desired": { "welcome": { "timestamp": 1620156892 }, "color": { "timestamp": 1620161643 } }, "reported": { "welcome": { "timestamp": 1620156892 }, "color": { "timestamp": 1620161643 } } }, "version": 10, "timestamp": 1620173908 } ``` 1. Enter a `desired` value in the terminal, such as `yellow`. The `shadow.py` sample app responds and displays the following messages in the terminal that show the change in the `reported` value to `yellow`. ``` Enter desired value: yellow Changed local shadow value to 'yellow'. Updating reported shadow value to 'yellow'... Update request published. Finished updating reported shadow value to 'yellow'. ``` In the **MQTT test client** in the **Amazon IoT console**, under **Subscriptions**, you see that the following topics received a message: + **\$1aws/things/*thingname*/shadow/update**: shows that both `desired` and `updated` values change to the color `yellow`. + **\$1aws/things/*thingname*/shadow/update/accepted**: shows the current values of the `desired` and `reported` states and their metadata and version information. + **\$1aws/things/*thingname*/shadow/update/documents**: shows the previous and current values of the `desired` and `reported` states and their metadata and version information. As the document **\$1aws/things/*thingname*/shadow/update/documents** also contains information that is contained in the other two topics, we can review it to see the state information. The previous state shows the reported value set to `green`, its metadata and version information, and the current state that shows the reported value updated to `yellow`. ``` { "previous": { "state": { "desired": { "welcome": "aws-iot", "color": "green" }, "reported": { "welcome": "aws-iot", "color": "green" } }, "metadata": { "desired": { "welcome": { "timestamp": 1617297888 }, "color": { "timestamp": 1617297898 } }, "reported": { "welcome": { "timestamp": 1617297888 }, "color": { "timestamp": 1617297898 } } }, "version": 10 }, "current": { "state": { "desired": { "welcome": "aws-iot", "color": "yellow" }, "reported": { "welcome": "aws-iot", "color": "yellow" } }, "metadata": { "desired": { "welcome": { "timestamp": 1617297888 }, "color": { "timestamp": 1617297904 } }, "reported": { "welcome": { "timestamp": 1617297888 }, "color": { "timestamp": 1617297904 } } }, "version": 11 }, "timestamp": 1617297904 } ``` 1. Now, if you enter another `desired` value, you see further changes to the `reported` values and message updates received by these topics. The version number also increments by 1. For example, if you enter the value `green`, the previous state reports the value `yellow` and the current state reports the value `green`. 1. **Edit Shadow document to observe delta events** To observe changes to the delta topic, edit the Shadow document in the Amazon IoT console. For example, you can change the `desired` value to the color `red`. To do this, in the Amazon IoT console, choose **Edit** and then set the `desired` value to red in the JSON, while keeping the `reported` value set to `green`. Before you save the change, keep the terminal open as you'll see the delta message reported in the terminal. ``` { "desired": { "welcome": "aws-iot", "color": "red" }, "reported": { "welcome": "aws-iot", "color": "green" } } ``` The `shadow.py` sample app responds to this change and displays messages in the terminal indicating the delta. In the MQTT test client, the `update` topics will have received a message showing changes to the `desired` and `reported` values. You also see that the topic **\$1aws/things/*thingname*/shadow/update/delta** received a message. To see the message, choose this topic, which is listed under **Subscriptions**. ``` { "version": 13, "timestamp": 1617318480, "state": { "color": "red" }, "metadata": { "color": { "timestamp": 1617318480 } } } ``` ## Step 3: Troubleshoot errors with Device Shadow interactions When you run the Shadow sample app, you might encounter issues with observing interactions with the Device Shadow service. If the program runs successfully and prompts you to enter a `desired` value, you should be able to observe the Device Shadow interactions by using the Shadow document and the MQTT test client as described previously. However, if you're unable to see the interactions, here are some things you can check: + **Check the thing name and its shadow in the Amazon IoT console** If you don't see the messages in the Shadow document, review the command and make sure it matches the thing name in the **Amazon IoT console**. You can also check whether you have a classic shadow by choosing your thing resource and then choosing **Shadows**. This tutorial focuses primarily on interactions with the classic shadow. You can also confirm that the device you used is connected to the internet. In the **Amazon IoT console**, choose the thing you created earlier, and then choose **Interact**. On the thing details page, you should see a message here that says: `This thing already appears to be connected.` + **Check the MQTT reserved topics you subscribed to** If you don't see the messages appear in the MQTT test client, check whether the topics you subscribed to are formatted correctly. MQTT Device Shadow topics have a format **\$1aws/things/*thingname*/shadow/** and might have `update`, `get`, or `delete` following it depending on actions you want to perform on the shadow. This tutorial uses the topic **\$1aws/things/*thingname*/shadow/\$1** so make sure you entered it correctly when subscribing to the topic in the **Topic filter** section of the test client. As you enter the topic name, make sure that the *thingname* is the same as the name of the Amazon IoT thing that you created earlier. You can also subscribe to additional MQTT topics to see if an update has been successfully performed. For example, you can subscribe to the topic **\$1aws/things/*thingname*/shadow/update/rejected** to receive a message whenever an update request failed so that you can debug connection issues. For more information about the reserved topics, see [Shadow topics](reserved-topics.md#reserved-topics-shadow) and [Device Shadow MQTT topics](device-shadow-mqtt.md). ## Step 4: Review the results and next steps **In this tutorial, you learned how to:** + Use the `shadow.py` sample app to specify desired states and update the shadow's current state. + Edit the Shadow document to observe delta events and how the `shadow.py` sample app responds to it. + Use the MQTT test client to subscribe to shadow topics and observe updates when you run the sample program. **Next steps** You can subscribe to additional MQTT reserved topics to observe updates to the shadow application. For example, if you only subscribe to the topic **\$1aws/things/*thingname*/shadow/update/accepted**, you'll see only the current state information when an update is successfully performed. You can also subscribe to additional shadow topics to debug issues or learn more about the Device Shadow interactions and also debug any issues with the Device Shadow interactions. For more information, see [Shadow topics](reserved-topics.md#reserved-topics-shadow) and [Device Shadow MQTT topics](device-shadow-mqtt.md). You can also choose to extend your application by using named shadows or by using additional hardware connected with the Raspberry Pi for the LEDs and observe changes to their state using messages sent from the terminal. For more information about the Device Shadow service and using the service in devices, apps, and services, see [Amazon IoT Device Shadow service](iot-device-shadows.md), [Using shadows in devices](device-shadow-comms-device.md), and [Using shadows in apps and services](device-shadow-comms-app.md).