Tutorial: Perform OTA updates on Espressif ESP32 using FreeRTOS Bluetooth Low Energy

1. Create an Amazon S3 bucket to store your update with versioning enabled to hold the firmware images.

2. Create an OTA Update service role and add the following managed policies to the role:

   • AWSIotLogging

   • AWSIotRuleActions

   • AWSIotThingsRegistration

   • AWSFreeRTOSOTAUpdate

3. Create a user that can perform OTA updates. This user can sign and deploy firmware updates to IoT devices in the account, and has access to do OTA updates on all devices. Access should be limited to trusted entities.

4. Follow the steps to Create an OTA user policy and attach it to your user.

1. Create an Amazon S3 bucket with versioning enabled to hold the firmware images.

2. Create a code-signing certificate that can be used to sign the firmware. Note the certificate Amazon Resource Name (ARN) when the certificate is imported.

   aws acm import-certificate --profile=ota-update-user --certificate file://ecdsasigner.crt --private-key file://ecdsasigner.key

   Example output:

   {
   "CertificateArn": "arn:aws:acm:us-east-1:<account>:certificate/<certid>"
   }

   You'll use the ARN later to create a signing profile. If you want, you can create the profile now with the following command:

   aws signer put-signing-profile --profile=ota-update-user --profile-name esp32Profile --signing-material certificateArn=arn:aws:acm:us-east-1:account:certificate/certid --platform AmazonFreeRTOS-Default --signing-parameters certname=/cert.pem

   Example output:

   {
   "arn": "arn:aws:signer::<account>:/signing-profiles/esp32Profile"
   }

Create an Amazon IoT policy

1. Sign in to the Amazon IoT console.

2. In the upper-right corner of the console, choose My Account. Under Account Settings, make a note of your 12-digit account ID.

3. In the left navigation pane, choose Settings. In Device data endpoint, make a note of the endpoint value. The endpoint should be something like xxxxxxxxxxxxxx.iot.us-west-2.amazonaws.com. In this example, the Amazon Region is "us-west-2".

4. In the left navigation pane, choose Secure, choose Policies, and then choose Create. If you don't have any policies in your account, you will see the message "You don’t have any policies yet" and you can choose Create a policy.

5. Enter a name for your policy, for example, "esp32_mqtt_proxy_iot_policy".

6. In the Add statements section, choose Advanced mode. Copy and paste the following JSON into the policy editor window. Replace aws-account-id with your account ID and aws-region with your Region (for example, "us-west-2").

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "iot:Connect",
         "Resource": "arn:aws:iot:aws-region:aws-account-id:*"
       },
       {
         "Effect": "Allow",
         "Action": "iot:Publish",
         "Resource": "arn:aws:iot:aws-region:aws-account-id:*"
       },
       {
         "Effect": "Allow",
         "Action": "iot:Subscribe",
         "Resource": "arn:aws:iot:aws-region:aws-account-id:*"
       },
       {
         "Effect": "Allow",
         "Action": "iot:Receive",
         "Resource": "arn:aws:iot:aws-region:aws-account-id:*"
       }
     ]
   }
   

7. Choose Create.

Create an Amazon IoT thing

1. Sign in to the Amazon IoT console.

2. In the left navigation pane, choose Manage, and then choose Things.

3. In the top-right corner, choose Create. If you don't have any things registered in your account, the message "You don’t have any things yet" is displayed and you can choose Register a thing.

4. On the Creating Amazon IoT things page, choose Create a single thing.

5. On the Add your device to the thing registry page, enter a name for your thing (for example, "esp32-ble"). Only alphanumeric, hyphen (-), and underscore (_) characters are allowed. Choose Next.

6. On the Add a certificate for your thing page, under Skip certificate and create thing, choose Create thing without certificate. Because we use the BLE proxy mobile app that uses an Amazon Cognito credential for authentication and authorization, no device certificate is required.

Create an Amazon Cognito App Client

1. Sign in to the Amazon Cognito console.

2. In the top-right navigation banner, choose Create a user pool.

3. Enter the pool name (for example, "esp32_mqtt_proxy_user_pool").

4. Choose Review defaults.

5. In App Clients, choose Add app client, and then choose Add an app client.

6. Enter an app client name (for example "mqtt_app_client").

7. Make sure Generate client secret is selected.

8. Choose Create app client.

9. Choose Return to pool details.

10. On the Review page of the user pool, choose Create pool. You should see a message that says "Your user pool was created successfully." Make a note of the pool ID.

11. In the navigation pane, choose App clients.

12. Choose Show Details. Make a note of the app client ID and the app client secret.

Create an Amazon Cognito identity pool

1. Sign in to the Amazon Cognito console.

2. Choose Create new identity pool.

3. Enter a name for the identity pool (for example, "mqtt_proxy_identity_pool").

4. Expand Authentication providers.

5. Choose the Cognito tab.

6. Enter the user pool ID and app client ID that you noted in previous steps.

7. Choose Create Pool.

8. On the next page, to create new roles for authenticated and unauthenticated identities, choose Allow.

9. Make a note of the identity pool ID, which is in the format us-east-1:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

Attach an IAM policy to the authenticated identity

1. Open the Amazon Cognito console.

2. Select the identity pool that you just created (for example, "mqtt_proxy_identity_pool").

3. Choose Edit identity pool.

4. Make a note of the IAM Role assigned to the authenticated role (for example, "Cognito_mqtt_proxy_identity_poolAuth_Role").

5. Open the IAM console.

6. In the navigation pane, choose Roles.

7. Search for the role assigned (for example, "Cognito_mqtt_proxy_identity_poolAuth_Role"), and then select it.

8. Choose Add inline policy, and then choose JSON.

9. Enter the following policy:

   {
          "Version": "2012-10-17",
          "Statement": [
          {
             "Effect": "Allow",
             "Action": [
                "iot:AttachPolicy",
                "iot:AttachPrincipalPolicy",
                "iot:Connect",
                "iot:Publish",
                "iot:Subscribe"
             ],
             "Resource": "*"
          }]
       }

10. Choose Review Policy.

11. Enter a policy name (for example, "mqttProxyCognitoPolicy").

12. Choose Create policy.

1. Download the latest version of the Amazon FreeRTOS code from the FreeRTOS GitHub repo.

2. To enable the OTA update demo, follow the steps in Getting started with the Espressif ESP32-DevKitC and the ESP-WROVER-KIT.

3. Make these additional modifications in the following files:

   1. Open vendors/espressif/boards/esp32/aws_demos/config_files/aws_demo_config.h and define CONFIG_OTA_UPDATE_DEMO_ENABLED.

   2. Open vendors/espressif/boards/esp32/aws_demos/common/config_files/aws_demo_config.h and change democonfigNETWORK_TYPES to AWSIOT_NETWORK_TYPE_BLE.

   3. Open demos/include/aws_clientcredential.h and enter your endpoint URL for clientcredentialMQTT_BROKER_ENDPOINT.

      Enter your thing name for clientcredentialIOT_THING_NAME (for example, "esp32-ble"). Certificates don't have to be added when you use Amazon Cognito credentials.

   4. Open vendors/espressif/boards/esp32/aws_demos/config_files/aws_iot_network_config.h and change configSUPPORTED_NETWORKS and configENABLED_NETWORKS to include only AWSIOT_NETWORK_TYPE_BLE.

   5. Open the vendors/vendor/boards/board/aws_demos/config_files/ota_demo_config.h file, and enter your certificate.

      #define otapalconfigCODE_SIGNING_CERTIFICATE [] = "your-certificate-key";

   The application should start up and print the demo version:

   11 13498 [iot_thread] [INFO ][DEMO][134980] Successfully initialized the demo. Network type for the demo: 2
   12 13498 [iot_thread] [INFO ][MQTT][134980] MQTT library successfully initialized.
   13 13498 [iot_thread] OTA demo version 0.9.20
   14 13498 [iot_thread] Creating MQTT Client...

1. Download the Android Bluetooth Low Energy SDK and a sample app from the amazon-freertos-ble-android-sdk GitHub repo.

2. Open the file app/src/main/res/raw/awsconfiguration.json and fill in the Pool Id, Region, AppClientId, and AppClientSecret using the instructions in the following JSON sample.

   {
     "UserAgent": "MobileHub/1.0",
     "Version": "1.0",
     "CredentialsProvider": {
       "CognitoIdentity": {
         "Default": {
           "PoolId": "Cognito->Manage Identity Pools->Federated Identities->mqtt_proxy_identity_pool->Edit Identity Pool->Identity Pool ID",
           "Region": "Your region (for example us-east-1)"
         }
       }
     },
   
     "IdentityManager": {
       "Default": {}
     },
   
     "CognitoUserPool": {
       "Default": {
         "PoolId": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> PoolId",
         "AppClientId": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> App clients ->Show Details",
         "AppClientSecret": "Cognito-> Manage User Pools -> esp32_mqtt_proxy_user_pool -> General Settings -> App clients ->Show Details",
         "Region": "Your region (for example us-east-1)"
       }
     }
   } 

3. Open app/src/main/java/software/amazon/freertos/DemoConstants.java and enter the policy name that you created earlier (for example, esp32_mqtt_proxy_iot_policy) and also the Region (for example, us-east-1).

4. Build and install the demo app.

   1. In Android Studio, choose Build, then Make Module app.

   2. Choose Run, then Run app. You can go to the logcat window pane in Android Studio to monitor log messages.

   3. On the Android device, create an account from the login screen.

   4. Create a user. If a user already exists, enter the credentials.

   5. Allow the Amazon FreeRTOS Demo to access the device’s location.

   6. Scan for Bluetooth Low Energy devices.

   7. Move the slider for the device found to On.

   8. Press y on the serial port debug console for the ESP32.

   9. Choose Pair & Connect.

5. The More... link becomes active after the connection is established. The connection state should change to "BLE_CONNECTED" in the Android device logcat when the connection is complete:

   2019-06-06 20:11:32.160 23484-23497/software.amazon.freertos.demo I/FRD: BLE connection state changed: 0; new state: BLE_CONNECTED

6. Before the messages can be transmitted, the Amazon FreeRTOS device and the Android device negotiate the MTU. You should see the following output in logcat:

   2019-06-06 20:11:46.720 23484-23497/software.amazon.freertos.demo I/FRD: onMTUChanged : 512 status: Success

7. The device connects to the app and starts sending MQTT messages using the MQTT proxy. To confirm that the device can communicate, make sure the MQTT_CONTROL characteristic data value changes to 01:

   2019-06-06 20:12:28.752 23484-23496/software.amazon.freertos.demo D/FRD: <-<-<- Writing to characteristic: MQTT_CONTROL with data: 01
   2019-06-06 20:12:28.839 23484-23496/software.amazon.freertos.demo D/FRD: onCharacteristicWrite for: MQTT_CONTROL; status: Success; value: 01

8. When the devices are paired, you will see a prompt on the ESP32 console. To enable BLE, press y. The demo won't function until you perform this step.

   E (135538) BT_GATT: GATT_INSUF_AUTHENTICATION: MITM Required
   W (135638) BT_L2CAP: l2cble_start_conn_update, the last connection update command still pending.
   E (135908) BT_SMP: Value for numeric comparison = 391840
   15 13588 [InputTask] Numeric comparison:391840
   16 13589 [InputTask] Press 'y' to confirm
   17 14078 [InputTask] Key accepted
   W (146348) BT_SMP: FOR LE SC LTK IS USED INSTEAD OF STK
   18 16298 [iot_thread] Connecting to broker...
   19 16298 [iot_thread] [INFO ][MQTT][162980] Establishing new MQTT connection.
   20 16298 [iot_thread] [INFO ][MQTT][162980] (MQTT connection 0x3ffd5754, CONNECT operation 0x3ffd586c) Waiting for operation completion.
   21 16446 [iot_thread] [INFO ][MQTT][164450] (MQTT connection 0x3ffd5754, CONNECT operation 0x3ffd586c) Wait complete with result SUCCESS.
   22 16446 [iot_thread] [INFO ][MQTT][164460] New MQTT connection 0x3ffc0ccc established.
   23 16446 [iot_thread] Connected to broker. 

1. To install the prerequisites, run the following commands:

   pip3 install boto3

   pip3 install pathlib

2. Increment the FreeRTOS application version in demos/include/aws_application_version.h.

3. Build a new .bin file.

4. Download the python script start_ota.py. To see the help contents for the script, run the following command in a terminal window:

   python3 start_ota.py -h

   You should see something like the following:

   usage: start_ota.py [-h] --profile PROFILE [--region REGION]
                       [--account ACCOUNT] [--devicetype DEVICETYPE] --name NAME
                       --role ROLE --s3bucket S3BUCKET --otasigningprofile
                       OTASIGNINGPROFILE --signingcertificateid
                       SIGNINGCERTIFICATEID [--codelocation CODELOCATION]
   Script to start OTA update
   optional arguments:
   -h, --help            show this help message and exit
   --profile PROFILE     Profile name created using aws configure
   --region REGION       Region
   --account ACCOUNT     Account ID
   --devicetype DEVICETYPE thing|group
   --name NAME           Name of thing/group
   --role ROLE           Role for OTA updates
   --s3bucket S3BUCKET   S3 bucket to store firmware updates
   --otasigningprofile OTASIGNINGPROFILE
                         Signing profile to be created or used
   --signingcertificateid SIGNINGCERTIFICATEID
                         certificate id (not arn) to be used
   --codelocation CODELOCATION
                         base folder location (can be relative) 

5. If you used the provided Amazon CloudFormation template to create resources, run the following command:

   python3 start_ota_stream.py --profile otausercf --name esp32-ble --role ota_ble_iot_role-sample --s3bucket afr-ble-ota-update-bucket-sample --otasigningprofile abcd --signingcertificateid certificateid

   You should see the update start in the ESP32 debug console:

   38 2462 [OTA Task] [prvParseJobDoc] Job was accepted. Attempting to start transfer.
   ---
   49 2867 [OTA Task] [prvIngestDataBlock] Received file block 1, size 1024
   50 2867 [OTA Task] [prvIngestDataBlock] Remaining: 1290
   51 2894 [OTA Task] [prvIngestDataBlock] Received file block 2, size 1024
   52 2894 [OTA Task] [prvIngestDataBlock] Remaining: 1289
   53 2921 [OTA Task] [prvIngestDataBlock] Received file block 3, size 1024
   54 2921 [OTA Task] [prvIngestDataBlock] Remaining: 1288
   55 2952 [OTA Task] [prvIngestDataBlock] Received file block 4, size 1024
   56 2953 [OTA Task] [prvIngestDataBlock] Remaining: 1287
   57 2959 [iot_thread] State: Active  Received: 5   Queued: 5   Processed: 5   Dropped: 0

6. When the OTA update is complete, the device restarts as required by the OTA update process. It then tries to connect using the updated firmware. If the upgrade has succeeded, the updated firmware is marked as active and you should see the updated version in the console:

   13 13498 [iot_thread] OTA demo version 0.9.21