Create a bulk import job (Amazon CLI) - Amazon IoT SiteWise
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Create a bulk import job (Amazon CLI)

Use the CreateBulkImportJob API operation to transfer data from Amazon S3 to Amazon IoT SiteWise. The CreateBulkImportJob API enables ingestion of large volumes of historical data, and buffered ingestion of analytical data streams in small batches. It provides a cost-effective primitive for data ingestion. The following example uses the Amazon CLI.

Important

Before creating a bulk import job, you must enable Amazon IoT SiteWise warm tier or Amazon IoT SiteWise cold tier. For more information, see Configure storage settings.

The CreateBulkImportJob API supports ingestion of historical data into Amazon IoT SiteWise with the option to set the adaptive-ingestion-flag parameter.

  • When set to false, the API ingests historical data without triggering computations or notifications.

  • When set to true, the API ingests new data, calculating metrics and transforming the data to optimize ongoing analytics and notifications within seven days.

Run the following command. Replace file-name with the name of the file that contains the bulk import job configuration.

aws iotsitewise create-bulk-import-job --cli-input-json file://file-name.json
Example Bulk import job configuration

The following are examples of configuration settings:

  • Replace adaptive-ingestion-flag with true or false.

    • If set to false, the bulk import job ingests historical data into Amazon IoT SiteWise.

    • If set to true, the bulk import job does the following:

      • Ingests new data into Amazon IoT SiteWise.

      • Calculates metrics and transforms, and supports notifications for data with a time stamp that's within seven days.

  • Replace delete-files-after-import-flag with true to delete the data from the S3 data bucket after ingesting into Amazon IoT SiteWise warm tier storage.

  • Replace error-bucket with the name of the Amazon S3 bucket to which errors associated with this bulk import job are sent.

  • Replace error-bucket-prefix with the prefix of the Amazon S3 bucket to which errors associated with this bulk import job are sent.

    Amazon S3 uses the prefix as a folder name to organize data in the bucket. Each Amazon S3 object has a key that is its unique identifier in the bucket. Each object in a bucket has exactly one key. The prefix must end with a forward slash (/). For more information, see Organizing objects using prefixes in the Amazon Simple Storage Service User Guide.

  • Replace data-bucket with the name of the Amazon S3 bucket from which data is imported.

  • Replace data-bucket-key with the key of the Amazon S3 object that contains your data. Each object has a key that is a unique identifier. Each object has exactly one key.

  • Replace data-bucket-version-id with the version ID to identify a specific version of the Amazon S3 object that contains your data. This parameter is optional.

  • Replace column-name with the column name specified in the .csv file.

  • Replace job-name with a unique name that identifies the bulk import job.

  • Replace job-role-arn with the IAM role that allows Amazon IoT SiteWise to read Amazon S3 data.

Note

Make sure that your role has the permissions shown in the following example. Replace data-bucket with the name of the Amazon S3 bucket that contains your data. Also, replace error-bucket with the name of the Amazon S3 bucket to which errors associated with this bulk import job are sent.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::data-bucket",
                "arn:aws:s3:::data-bucket/*",
            ],
            "Effect": "Allow"
        },
		    {
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::error-bucket",
                "arn:aws:s3:::error-bucket/*"
            ],
            "Effect": "Allow"
        }
    ]
}
{
   "adaptiveIngestion": adaptive-ingestion-flag,
   "deleteFilesAfterImport": delete-files-after-import-flag,       
   "errorReportLocation": { 
      "bucket": "error-bucket",
      "prefix": "error-bucket-prefix"
   },
   "files": [ 
      { 
         "bucket": "data-bucket",
         "key": "data-bucket-key",
         "versionId": "data-bucket-version-id"
      }
   ],
   "jobConfiguration": { 
      "fileFormat": { 
         "csv": { 
            "columnNames": [ "column-name" ]
         }
      }
   },
   "jobName": "job-name",
   "jobRoleArn": "job-role-arn"    
}
Example response
{
   "jobId":"f8c031d0-01d1-4b94-90b1-afe8bb93b7e5",
   "jobStatus":"PENDING",
   "jobName":"myBulkImportJob"
}