Using a Docker container

This section includes information about how to build your own Docker container. There is a security risk if you re-use Docker containers built by third parties: these containers can execute arbitrary code with your user permissions. Make sure you trust the author of any third-party container before using it.

Here are the steps you would take to set up periodic data analysis on data which has arrived since the last analysis was performed:

With the SQL query and delta window to generate the dataset, and the container with your application, Amazon IoT Analytics creates a scheduled production dataset that runs at the interval you specify on data from the delta window, producing your desired output and sending notifications.

You can pause your production dataset application and resume it whenever you choose to do so. When you resume your production dataset application, Amazon IoT Analytics, by default, catches up all the data that has arrived since last execution, but hasn't been analyzed yet. You can also configure how you want to resume your production dataset job window length) by performing a series of consecutive runs. Alternatively, you can resume your production dataset application by capturing only the newly arrived data that fits within the specified size of your delta window.

Please note the following limitations when creating or defining a dataset which is triggered by the creation of another dataset:

The following errors may be returned when creating a container dataset which is triggered by a SQL dataset:

Custom Docker container input/output variables

This section demonstrates how the program which is run by your custom Docker image may read input variables and upload its output.

Params File

The input variables and the destinations to which you want to upload output are stored in a JSON file located at /opt/ml/input/data/iotanalytics/params on the instance that executes your docker image. Here is an example of the contents of that file.

{
   "Context": {
       "OutputUris": {
           "html": "s3://aws-iot-analytics-dataset-xxxxxxx/notebook/results/iotanalytics-xxxxxxx/output.html",
           "ipynb": "s3://aws-iot-analytics-dataset-xxxxxxx/notebook/results/iotanalytics-xxxxxxx/output.ipynb"
       }
   },
   "Variables": {
       "source_dataset_name": "mydataset",
       "source_dataset_version_id": "xxxx",
       "example_var": "hello world!",
       "custom_output": "s3://aws-iot-analytics/dataset-xxxxxxx/notebook/results/iotanalytics-xxxxxxx/output.txt"
   }
}

In addition to the name and version ID of your dataset, the Variables section contains the variables specified in the iotanalytics:CreateDataset invocation-- in this example, a variable example_var was given the value hello world!. A custom output URI was also provided in the custom_output variable. The OutputUris field contains default locations to which the container can upload its output-- in this example, default output URIs were provided for both ipynb and html output.

Input variables

The program launched by your Docker image can read variables from the params file. Here is an example program which opens the params file, parses it, and prints the value of the example_var variable.

import json

with open("/opt/ml/input/data/iotanalytics/params") as param_file:
    params = json.loads(param_file.read())
example_var = params["Variables"]["example_var"]
print(example_var)

Uploading output

The program launched by your Docker image might also store its output in an Amazon S3 location. The output must be loaded with a "bucket-owner-full-control" access control list. The access list grants the Amazon IoT Analytics service control over the uploaded output. In this example we extend the previous one to upload the contents of example_var to the Amazon S3 location defined by custom_output in the params file.

import boto3
import json
from urllib.parse import urlparse

ACCESS_CONTROL_LIST = "bucket-owner-full-control"

with open("/opt/ml/input/data/iotanalytics/params") as param_file:
    params = json.loads(param_file.read())
example_var = params["Variables"]["example_var"]

outputUri = params["Variables"]["custom_output"]
# break the S3 path into a bucket and key
bucket = urlparse(outputUri).netloc
key = urlparse(outputUri).path.lstrip("/")

s3_client = boto3.client("s3")
s3_client.put_object(Bucket=bucket, Key=key, Body=example_var, ACL=ACCESS_CONTROL_LIST)