Exporting data to Amazon S3 by using asset property notifications - Amazon IoT SiteWise
Creating the Amazon CloudFormation stackViewing your data in Amazon S3
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Exporting data to Amazon S3 by using asset property notifications

You can export incoming data from Amazon IoT SiteWise to an Amazon S3 bucket in your account. You can back up your data in a format that you can use to create historical reports or to analyze your data with complex methods.

Note

Amazon IoT SiteWise also supports cold tier storage that let you save data in a customer-managed Amazon S3 bucket. For more information about supported storage tiers, see Managing data storage.

Amazon IoT SiteWise provides this feature as an Amazon CloudFormation template. When you create a stack from the template, Amazon CloudFormation creates the required Amazon resources to stream incoming data from Amazon IoT SiteWise to an S3 bucket.

Then, the S3 bucket receives all of your asset property data sent from Amazon IoT SiteWise property value update messages. The S3 bucket also receives your asset metadata, which includes asset and property names and other information.

For more information about how to enable property value update messages for the asset properties to export to Amazon S3, see Interacting with other Amazon services.

This feature stores your asset property data and asset metadata in the Apache Parquet format in Amazon S3. Parquet is a columnar data format that saves space and enables faster queries compared to row-oriented formats like JSON.

Note

When this feature retrieves asset metadata, it supports up to approximately 1,500 assets. This limitation applies only to asset metadata. This limitation doesn't apply to the number of assets supported when the feature exports asset property data.

Each resource's name includes a prefix that you can customize when you create the stack. Resources include the following:

  • An Amazon S3 bucket

  • Amazon Lambda functions

  • An Amazon IoT Core rule

  • Amazon Identity and Access Management roles

  • An Amazon Kinesis Data Firehose stream

  • An Amazon Glue database

For a complete list, see Resources created from the template.

Important

You will be charged for the resources that this Amazon CloudFormation template creates and consumes. These charges include data storage and data transfer for multiple Amazon services.

Topics

Creating the Amazon CloudFormation stack

You can create a stack in Amazon CloudFormation to export your asset data to Amazon S3.

To export data to Amazon S3

  1. Open the Amazon CloudFormation template and sign in to the Amazon Web Services Management Console.

  2. On the Create stack page, choose Next at the bottom of the page.

  3. On the Specify stack details page, enter a BucketName for the S3 bucket that this template creates in order to receive asset data. This bucket name must be globally unique. For more information, see Rules for bucket naming in the Amazon Simple Storage Service User Guide.

  4. (Optional) Change any of the template's other parameters:

    • GlobalResourcePrefix – A prefix for names of global resources, such as IAM roles, created from this template.

    • LocalResourcePrefix – A prefix for names of resources created from this template in the current Region.

    Note

    If you create this template multiple times, you should change the bucket name and resource prefix parameters in order to avoid resource name conflicts.

  5. Choose Next.

  6. On the Configure stack options page, choose Next.

  7. At the bottom of the page, select the check box that says I acknowledge that Amazon CloudFormation might create IAM resources.

  8. Choose Create stack.

    The stack takes a few minutes to create. If the stack fails to create, your account might have insufficient permissions, or you might have entered a bucket name that already exists. Use the following steps to delete the stack and try again:

    1. Choose Delete in the upper-right corner.

      The stack takes a few minutes to delete.

      Note

      Amazon CloudFormation doesn't delete S3 buckets or CloudWatch log groups. You can delete these resources in the consoles for those services.

    2. If the stack fails to delete, choose Delete again.

    3. If the stack fails to delete again, follow the steps in the Amazon CloudFormation console to skip the resources that failed to delete, and try again.

  9. After the Amazon CloudFormation stack creates successfully, follow the next procedure to explore your asset property data in Amazon S3.

Important

After you create the stack, you can see the new resources in your Amazon account. The feature might stop working correctly if you delete or modify these resources. We recommend that you don't modify these resources unless you want to stop sending data to the bucket or want to customize this feature.

Viewing your data in Amazon S3

After you create the feature, you can view your asset property data and asset metadata in Amazon S3.

Note

Asset metadata updates every six hours. You might need to wait up to six hours to see asset metadata appear in the S3 bucket.

This feature stores asset property data in the following columns, where each row contains a data point:

  • type – The type of property notification (PropertyValueUpdate).

  • asset_id – The ID of the asset that received a data point.

  • asset_property_id – The ID of the property that received a data point for the asset.

  • time_in_seconds – The time at which the data was received, expressed in seconds in Unix epoch time.

  • offset_in_nanos – The nanosecond offset from timeInSeconds.

  • asset_property_quality – The quality of the data point: GOOD, UNCERTAIN, or BAD.

  • asset_property_value – The value of the data point.

  • asset_property_data_type – The data type of the asset property: boolean, double, integer, or string.

This feature stores asset metadata in the following columns, where each row contains an asset property:

  • asset_id – The ID of the asset.

  • asset_name – The name of the asset.

  • asset_model_id – The ID of the asset's model.

  • asset_property_id – The ID of the asset property.

  • asset_property_name – The name of the asset property.

  • asset_property_data_type – The data type of the asset property: BOOLEAN, DOUBLE, INTEGER, or STRING.

  • asset_property_unit – The unit of the asset property.

  • asset_property_alias – The alias of the asset property.

To view your Amazon IoT SiteWise data in Amazon S3

  1. Navigate to the Amazon S3 console.

  2. From the list of buckets, choose the bucket with the name you chose when you created the template.

  3. In the bucket, choose one of the following folders:

    • asset-property-updates – This folder contains asset property data exported from Amazon IoT SiteWise.

    • asset-metadata – This folder contains asset details exported from Amazon IoT SiteWise.

  4. Choose the object that you want to view.

  5. On the object's page, do the following:

    1. Choose the Select from tab.

      In this panel, you can preview records from Parquet files.

    2. For File format, choose Parquet.

    3. To show the contents of the file in JSON format, choose Show file preview.

Note

If new data doesn't appear in the bucket, check that you enabled property value update notifications for your asset properties. For more information, see Interacting with other Amazon services.

For more information about how to analyze your asset data stored in the S3 bucket, see Analyzing exported data with Amazon Athena.