# Add an Amazon IoT SiteWise buffered destination using Amazon S3 The buffered destination type allows you to save on ingestion costs into Amazon IoT SiteWise if you don't need the data in real-time. It enables you to temporarily store your IoT data in an Amazon S3 bucket before importing it into Amazon IoT SiteWise. Or, you can simply upload your data to S3 for storage, regardless of whether you plan to import it to Amazon IoT SiteWise. This is useful for batching and buffering data from your devices and gateways before ingesting it into Amazon IoT SiteWise. With this option, data is uploaded to the specified S3 bucket in Parquet format at a configured frequency. You can then import this data into Amazon IoT SiteWise storage for further analysis and processing. **To add a destination buffered using Amazon S3** Use the Amazon IoT SiteWise console or Amazon CLI to add a destination that buffers data using Amazon S3 to your SiteWise Edge gateway. ------ #### [ Console ] Use the Amazon Web Services Management Console to add an Amazon IoT SiteWise destination buffered using Amazon S3. 1. Open the [Amazon IoT SiteWise console](https://console.amazonaws.cn/iotsitewise/). 1. In the navigation pane, choose **Edge gateways**. 1. Select the gateway to which you want to add a destination. 1. In the **Destinations** section, choose **Add destination**. 1. On the **Add destination** page, enter **Destination details**: 1. A name for your destination in the **Destination name** field. 1. Select **Amazon IoT SiteWise buffered using Amazon S3** for **Destination type**. Amazon IoT SiteWise buffered using Amazon S3 sends data to Amazon Simple Storage Service in batches, in Parquet format, and then imports the data into Amazon IoT SiteWise storage. 1. Enter the Amazon S3 URL for the location where you want to store your gateway data. You can browse for the path by choosing **Browse S3**. Once a bucket is added, you can also view the bucket by choosing **View**. 1. Specify how often your gateway should upload data to Amazon S3 by entering a time frame and selecting a time increment for **Data upload frequency**. The frequency value should be greater than 0 and less than or equal to 30 days. 1. In **Data storage settings**, determine what to do with your gateway data after importing it to Amazon IoT SiteWise. There are two decisions to make regarding data storage: + If you want to copy imported data into Amazon IoT SiteWise storage, select the **Copy data to storage** check box. This option duplicates the imported data from your configured Amazon S3 bucket into Amazon IoT SiteWise storage. + If you choose to import your data from your Amazon S3 bucket into Amazon IoT SiteWise storage, you can also specify whether the imported data should be deleted after the import is complete. Select the **Delete data from Amazon S3** check box to delete the imported date from the configured Amazon S3 bucket after importing it to Amazon IoT SiteWise storage. 1. Add path filters to your destination. For more information see, [Add path filters to Amazon IoT SiteWise Edge destinations](destinations-add-path-filters.md). ------ #### [ Amazon CLI ] **Example : Create a new Amazon IoT SiteWise destination buffered using Amazon S3** Use the [UpdateGatewayCapabilityConfiguration](https://docs.amazonaws.cn/iot-sitewise/latest/APIReference/API_UpdateGatewayCapabilityConfiguration.html) API to configure the publisher. Set the `capabilityNamespace` parameter to `iotsitewise:publisher:3`. ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "type": "SITEWISE_BUFFERED", "name": "your-s3-destination-name", "config": { "targetBucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/Optional/SomeFolder", "publishPolicy": { "publishFrequency": "15m", "localSizeLimitGB": 10 }, "siteWiseImportPolicy": { "enableSiteWiseStorageImport": true, "enableDeleteAfterImport": true, "bulkImportJobRoleArn": "arn:aws:iam::123456789012:role/your-role-name" } }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` **Example : Update an Amazon IoT SiteWise destination buffered using Amazon S3** To update an existing Amazon IoT SiteWise real-time destination, first use the `DescribeGatewayCapabilityConfiguration` API to find the `destinationId`. The publisher namespace: `iotsitewise:publisher:3` ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "id": "your-existing-destination-id", "type": "SITEWISE_BUFFERED", "name": "your-s3-destination-name", "config": { "targetBucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/Optional/SomeFolder", "publishPolicy": { "publishFrequency": "15m", "localSizeLimitGB": 10 }, "siteWiseImportPolicy": { "enableSiteWiseStorageImport": true, "enableDeleteAfterImport": true, "bulkImportJobRoleArn": "arn:aws:iam::123456789012:role/your-role-name" } }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` The following configuration options are specific to MQTT-enabled gateways using the `iotsitewise:publisher:3` namespace. `sources` Defines data sources to transfer of data from your industrial equipment to Amazon IoT SiteWise. For MQTT-enabled gateways, use `MQTT`. Type: Array of objects Required: Yes `destinations` Defines where to send data. Destinations are either real-time or buffered using Amazon S3. At least one destination object is required, but you can add an empty array. You can have one real-time destination for each gateway. For more information, see [Understand Amazon IoT SiteWise Edge destinations](gw-destinations.md#source-destination). Type: Array of objects Required: Yes `id` The unique identifier for the destination. You can either provide an existing destination ID or leave it blank to have a new ID automatically generated for the destination. Type: String Required: No `type` Type of destination. Options include: `SITEWISE_REALTIME` and `SITEWISE_BUFFERED`. Choose `SITEWISE_BUFFERED`. + `SITEWISE_REALTIME` (default) – Send data directly to Amazon IoT SiteWise storage in real-time. For more information, see [Add an Amazon IoT SiteWise Edge real-time destination](destinations-real-time.md). + `SITEWISE_BUFFERED` – Send data to Amazon S3 in batches in Parquet format, and then import into Amazon IoT SiteWise storage. Type: String Required: Yes `name` A unique name for the destination. Type: String Required: Yes `config` Configuration specific to the destination type in JSON format. The configuration varies between real-time and buffered destinations. Type: Object Required: Yes `targetBucketArn` The bucket ARN to publish to. Choose the same Amazon Web Services Region for both Amazon IoT SiteWise and Amazon S3. If a prefix is chosen, it must have between 1-255 characters. Amazon IoT SiteWise, including the gateway, will have access to the entire specified S3 bucket. We recommend using a dedicated bucket for buffered data ingestion. Type: String Required: Yes `publishPolicy` Details of the publishing policy. Type: Object Required: Yes `publishFrequency` The frequency with which the SiteWise Edge gateway publishes to the Amazon S3 bucket. Data upload frequency to Amazon S3 must be greater than 0 minutes and less than or equal to 30 days. You can use `m`, `h`, and `d` when you specify a publishing frequency age. Note that `m` represents minutes, `h` represents hours, and `d` represents days. The default value is 15 minutes. Type: String Required: Yes `localSizeLimitGB` The maximum size of the files written to local disk in GB. If this threshold is breached, the publisher publishes all buffered data to its destination. Type: Integer Required: Yes `siteWiseImportPolicy` Details of the import policy for importing data to Amazon IoT SiteWise. Type: Object Required: Yes `enableSiteWiseStorageImport` Set this to `true` to import data from an Amazon S3 bucket to Amazon IoT SiteWise storage. It initially makes a copy of the data in Amazon IoT SiteWise. Then, if you set `enableDeleteAfterImport` to true, the data in S3 deletes after copying to Amazon IoT SiteWise. Pricing implications apply. The default value is `true`. Type: Boolean Required: Yes `enableDeleteAfterImport` Set this to `true` to delete the file in the Amazon S3 bucket after ingestion into the Amazon IoT SiteWise storage. The default value is `true`. Type: Boolean Required: Yes `bulkImportJobRoleArn` The ARN of the IAM role that Amazon IoT SiteWise assumes to read buffered data from Amazon S3 during data ingestion. This role is used when an edge device calls on Amazon IoT SiteWise APIs to initiate the bulk import process. If `enableSiteWiseStorageImport` is set to `true`, this parameter is required. Type: String Required: No ------ Add path filters for your destination. For more information, see [Add path filters to Amazon IoT SiteWise Edge destinations](destinations-add-path-filters.md).