Developing Amazon IoT TwinMaker time-series data connectors - Amazon IoT TwinMaker
Amazon IoT TwinMaker time-series data connector prerequisitesTime-series data connector backgroundDeveloping a time-series data connectorImproving your data connectorTesting your connectorSecurityCreating Amazon IoT TwinMaker resources What's next
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Developing Amazon IoT TwinMaker time-series data connectors

This section explains how to develop a time-series data connector in a step-by-step process. Additionally, we present an example time-series data connector based of the entire cookie factory sample, which includes 3D models, entities, components, alarms, and connectors. The cookie factory sample source is available on the Amazon IoT TwinMaker samples GitHub repository .

Topics

Amazon IoT TwinMaker time-series data connector prerequisites

Before developing your time-series data connector, we recommend that you complete the following tasks:

Note

For an example of a fully implemented connector, see our cookie factory example implementation.

Time-series data connector background

Imagine you are working with a factory that has a set of cookie mixers and a water tank. You would like to build Amazon IoT TwinMaker digital twins of these physical entities so that you can monitor their operational states by checking various time-series metrics.

You have on-site sensors set up and you are already streaming measurement data into a Timestream database. You want to be able to view and organize the measurement data in Amazon IoT TwinMaker with minimal overhead. You can accomplish this task by using a time-series data connector. The following image shows an example telemetry table, which is populated through the use of a time-series connector.

An example of telemetry table data that includes the Asset ID, Type, measure, time, and values.

The datasets and the Timestream table used in this screenshot are available in the Amazon IoT TwinMaker samples GitHub repository. Also see the cookie factory example connector for the implementation, which produces the result shown in the preceding screenshot.

Time-series data connector data flow

For data plane queries, Amazon IoT TwinMaker fetches the corresponding properties of both components and component types from components and component types definitions. Amazon IoT TwinMaker forwards properties to Amazon Lambda functions along with any API query parameters in the query.

Amazon IoT TwinMaker uses Lambda functions to access and resolve queries from data sources and return the results of those queries. The Lambda functions use the component and component type properties from the data plane to resolve the initial request.

The results of the Lambda query are mapped to an API response and returned to you.

Amazon IoT TwinMaker defines the data connector interface and uses that to interact with Lambda functions. Using data connectors, you can query your data source from Amazon IoT TwinMaker API without any data migration efforts. The following image outlines the basic data flow described in the previous paragraphs.

API requests and responses use 3P Connector requests and responses that access a data source.

Developing a time-series data connector

The following procedure outlines a development model that incrementally builds up to a functional time-series data connector. The basic steps are as follows:

  1. Create a valid basic component type

    In a component type, you define common properties that are shared across your components. To learn more about defining component types, see Using and creating component types.

    Amazon IoT TwinMaker uses an entity-component modeling pattern so each component is attached to an entity. We recommend that you model each physical item as an entity and model different data sources with their own component types.

    The following example shows a Timestream template component type with one property:

    {"componentTypeId": "com.example.timestream-telemetry",
    "workspaceId": "MyWorkspace",
    "functions": {
        "dataReader": {
            "implementedBy": {
                "lambda": {
                    "arn": "lambdaArn"
                }
            }
        }
    },
    "propertyDefinitions": {
        "telemetryType": {
            "dataType": { "type": "STRING" },
            "isExternalId": false,
            "isStoredExternally": false,
            "isTimeSeries": false,
            "isRequiredInEntity": true
        },
        "telemetryId": {
            "dataType": { "type": "STRING" },
            "isExternalId": true,
            "isStoredExternally": false,
            "isTimeSeries": false,
            "isRequiredInEntity": true
        },
        "Temperature": {
            "dataType": { "type": "DOUBLE" },
            "isExternalId": false,
            "isTimeSeries": true,
            "isStoredExternally": true,
            "isRequiredInEntity": false
        }
    }
}

    The key elements of the component type are the following:

    • The telemetryId property identifies the unique key of the physical item in the corresponding data source. The data connector uses this property as a filter condition to only query values associated with the given item. Additionally, if you include the telemetryId property value in the data plane API response, then the client side takes the ID and can perform a reverse lookup if necessary.

    • The lambdaArn field identifies the Lambda function with which the component type engages.

    • The isRequiredInEntity flag enforces the ID creation. This flag is required so that when the component is created, the item's ID is also instantiated.

    • The TelemetryId is added to the component type as an external id so that the item can be identified in the Timestream table.

  2. Create a component with the component type

    To use the component type you created, you must create a component and attach it to the entity from which you wish to retrieve data. The following steps detail the process of creating that component:

    1. Navigate to the Amazon IoT TwinMaker console.

    2. Select and open the same workspace in which you created the component types.

    3. Navigate to the entity page.

    4. Create a new entity or select an existing entity from the table.

    5. Once you have selected the entity you wish to use, choose Add component to open the Add component page.

    6. Give the component a name and for the Type, choose the component type you created with the template in 1. Create a valid basic component type.

  3. Make your component type call a Lambda connector

    The Lambda connector needs to access the data source and generate the query statement based on the input and forward it to the data source. The following example shows a JSON request template that does this. 

    {
  "workspaceId": "MyWorkspace",
  "entityId": "MyEntity",
  "componentName": "TelemetryData",
  "selectedProperties": ["Temperature"],
  "startTime": "2022-08-25T00:00:00Z",
  "endTime": "2022-08-25T00:00:05Z",
  "maxResults": 3,
  "orderByTime": "ASCENDING",
  "properties": {
      "telemetryType": {
          "definition": {
              "dataType": { "type": "STRING" },
              "isExternalId": false,
              "isFinal": false,
              "isImported": false,
              "isInherited": false,
              "isRequiredInEntity": false,
              "isStoredExternally": false,
              "isTimeSeries": false
          },
          "value": {
              "stringValue": "Mixer"
          }
      },
      "telemetryId": {
          "definition": {
              "dataType": { "type": "STRING" },
              "isExternalId": true,
              "isFinal": true,
              "isImported": false,
              "isInherited": false,
              "isRequiredInEntity": true,
              "isStoredExternally": false,
              "isTimeSeries": false
          },
          "value": {
              "stringValue": "item_A001"
          }
      },
      "Temperature": {
          "definition": {
              "dataType": { "type": "DOUBLE", },
              "isExternalId": false,
              "isFinal": false,
              "isImported": true,
              "isInherited": false,
              "isRequiredInEntity": false,
              "isStoredExternally": false,
              "isTimeSeries": true
          }
      }
  }
}

    The key elements of the request:

    • The selectedProperties is a list you populate with the properties for which you want Timestream measurements.

    • The startDateTime, startTime, EndDateTime, and endTime fields specify a time range for the request. This determines the sample range for the measurements returned.

    • The entityId is the name of the entity from which you are querying data.

    • The componentName is the name of the component from which you are querying data.

    • Use the orderByTime field to organize the order in which the results are displayed.

    In the preceding example request, we would expect to get a series of samples for the selected properties during the given time window for the given item, with the selected time order. The response statement can be summarized as the following:

    {
  "propertyValues": [
    {
      "entityPropertyReference": {
        "entityId": "MyEntity",
        "componentName": "TelemetryData",
        "propertyName": "Temperature"
      },
      "values": [
        {
          "time": "2022-08-25T00:00:00Z",
          "value": {
            "doubleValue": 588.168
          }
        },
        {
          "time": "2022-08-25T00:00:01Z",
          "value": {
            "doubleValue": 592.4224
          }
        },
        {
          "time": "2022-08-25T00:00:02Z",
          "value": {
            "doubleValue": 594.9383
          }
        }
      ]
    }
  ],
  "nextToken": "..."
}

  4. Update your component type to have two properties

    The following JSON template shows a valid component type with two properties:

    {
    "componentTypeId": "com.example.timestream-telemetry",
    "workspaceId": "MyWorkspace",
    "functions": {
        "dataReader": {
            "implementedBy": {
                "lambda": {
                    "arn": "lambdaArn"
                }
            }
        }
    },
    "propertyDefinitions": {
        "telemetryType": {
            "dataType": { "type": "STRING" },
            "isExternalId": false,
            "isStoredExternally": false,
            "isTimeSeries": false,
            "isRequiredInEntity": true
        },
        "telemetryId": {
            "dataType": { "type": "STRING" },
            "isExternalId": true,
            "isStoredExternally": false,
            "isTimeSeries": false,
            "isRequiredInEntity": true
        },
        "Temperature": {
            "dataType": { "type": "DOUBLE" },
            "isExternalId": false,
            "isTimeSeries": true,
            "isStoredExternally": true,
            "isRequiredInEntity": false
        },
        "RPM": {
            "dataType": { "type": "DOUBLE" },
            "isExternalId": false,
            "isTimeSeries": true,
            "isStoredExternally": true,
            "isRequiredInEntity": false
        }
    }
}

  5. Update the Lambda connector to handle the second property

    The Amazon IoT TwinMaker data plane API supports querying multiple properties in a single request, and Amazon IoT TwinMaker follows a single request to a connector by providing a list of selectedProperties.

    The following JSON request shows a modified template that now supports a request for two properties.

    {
  "workspaceId": "MyWorkspace",
  "entityId": "MyEntity",
  "componentName": "TelemetryData",
  "selectedProperties": ["Temperature", "RPM"],
  "startTime": "2022-08-25T00:00:00Z",
  "endTime": "2022-08-25T00:00:05Z",
  "maxResults": 3,
  "orderByTime": "ASCENDING",
  "properties": {
      "telemetryType": {
          "definition": {
              "dataType": { "type": "STRING" },
              "isExternalId": false,
              "isFinal": false,
              "isImported": false,
              "isInherited": false,
              "isRequiredInEntity": false,
              "isStoredExternally": false,
              "isTimeSeries": false
          },
          "value": {
              "stringValue": "Mixer"
          }
      },
      "telemetryId": {
          "definition": {
              "dataType": { "type": "STRING" },
              "isExternalId": true,
              "isFinal": true,
              "isImported": false,
              "isInherited": false,
              "isRequiredInEntity": true,
              "isStoredExternally": false,
              "isTimeSeries": false
          },
          "value": {
              "stringValue": "item_A001"
          }
      },
      "Temperature": {
          "definition": {
              "dataType": { "type": "DOUBLE" },
              "isExternalId": false,
              "isFinal": false,
              "isImported": true,
              "isInherited": false,
              "isRequiredInEntity": false,
              "isStoredExternally": false,
              "isTimeSeries": true
          }
      },
      "RPM": {
          "definition": {
              "dataType": { "type": "DOUBLE" },
              "isExternalId": false,
              "isFinal": false,
              "isImported": true,
              "isInherited": false,
              "isRequiredInEntity": false,
              "isStoredExternally": false,
              "isTimeSeries": true
          }
      }
  }
}

    Similarly, the corresponding response is also updated, as shown in the following example:

    {
  "propertyValues": [
    {
      "entityPropertyReference": {
        "entityId": "MyEntity",
        "componentName": "TelemetryData",
        "propertyName": "Temperature"
      },
      "values": [
        {
          "time": "2022-08-25T00:00:00Z",
          "value": {
            "doubleValue": 588.168
          }
        },
        {
          "time": "2022-08-25T00:00:01Z",
          "value": {
            "doubleValue": 592.4224
          }
        },
        {
          "time": "2022-08-25T00:00:02Z",
          "value": {
            "doubleValue": 594.9383
          }
        }
      ]
    },
    {
      "entityPropertyReference": {
        "entityId": "MyEntity",
        "componentName": "TelemetryData",
        "propertyName": "RPM"
      },
      "values": [
        {
          "time": "2022-08-25T00:00:00Z",
          "value": {
            "doubleValue": 59
          }
        },
        {
          "time": "2022-08-25T00:00:01Z",
          "value": {
            "doubleValue": 60
          }
        },
        {
          "time": "2022-08-25T00:00:02Z",
          "value": {
            "doubleValue": 60
          }
        }
      ]
    }
  ],
  "nextToken": "..."
}
    Note

    In terms of the pagination for this case, the page size in the request applies to all properties. This means that with five properties in the query and a page size of 100, if there are enough data points in the source, you should expect to see 100 data points per property, with 500 data points in total.

    For an example implementation, see Snowflake connector sample on GitHub.

Improving your data connector

Handling exceptions

It is safe for the Lambda connector to throw exceptions. In the data plane API call, the Amazon IoT TwinMaker service waits for the Lambda function to return a response. If the connector implementation throws an exception, Amazon IoT TwinMaker translates the exception type to be ConnectorFailure, making the API client aware that an issue happened inside the connector.

Handling pagination

In the example, Timestream provides a utility function which can help support pagination natively. However, for some other query interfaces, such as SQL, it might need extra effort to implement an efficient pagination algorithm. There is a Snowflake connector example that handles pagination in an SQL interface.

When the new token is returned to Amazon IoT TwinMaker through the connector response interface, the token is encrypted before being returned to the API client. When the token is included in another request, Amazon IoT TwinMaker decrypts it before forwarding it to the Lambda connector. We recommend that you avoid adding sensitive information to the token.

Testing your connector

Though you can still update the implementation after you link the connector to the component type, we strongly recommend you verify the Lambda connector before integrating with Amazon IoT TwinMaker.

There are multiple ways to test your Lambda connector: you can test the Lambda connector in the Lambda console or locally in the Amazon CDK.

For more information on testing your Lambda functions, see Testing Lambda functions and Locally testing Amazon CDK applications.

Security

For documentation on security best practices with Timestream, see Security in Timestream.

For an example of SQL injection prevention, see the following Python script in Amazon IoT TwinMaker Samples GitHub Repository.

Creating Amazon IoT TwinMaker resources

Once you have implemented the Lambda function, you can create Amazon IoT TwinMaker resources such as component types, entities, and components through the Amazon IoT TwinMaker console or API.

Note

If you follow the setup instructions in the GitHub sample, all Amazon IoT TwinMaker resources are available automatically. You can check the component type definitions in the Amazon IoT TwinMaker GitHub sample. Once the component type is used by any components, the property definitions and functions of the component type cannot be updated.

Integration testing

We recommend having an integrated test with Amazon IoT TwinMaker to verify the data plane query works end-to-end. You can perform that through GetPropertyValueHistory API or easily in Amazon IoT TwinMaker console.

A TwinMaker Component information console page shows the component's name, type, status, and so on.

In the Amazon IoT TwinMaker console, go to component details and then under the Test, you’ll see all the properties in the component are listed there. The Test area of the console allows you to test time-series properties as well as non-time-series properties. For time-series properties you can also use the GetPropertyValueHistory API and for non-time-series properties use GetPropertyValue API. If your Lambda connector supports multiple property query, you can choose more than one property.

A portion of a TwinMaker Component information console page showing a component's test.

What's next

You can now set up an Amazon IoT TwinMaker Grafana dashboard to visualize metrics. You can also explore other data connector samples in the Amazon IoT TwinMaker samples GitHub repository to see if they fit your use case.