Performing bulk import and export operations
This topic covers how to perform bulk import and export operations and how to handle errors in your transfer jobs. It provides examples of transfer jobs using CLI commands.
The Amazon IoT TwinMaker API Reference contains information on the CreateMetadataTransferJob and other API actions.
Topics
metadataTransferJob prerequisites
Please complete the following prerequisites before you run a metadataTransferJob:
Create an Amazon IoT TwinMaker workspace. The workspace can be the import destination or export source for a metadataTransferJob. For information on creating a workspace see, Create a workspace.
Create an Amazon S3 bucket to store resources. For more information on using Amazon S3 see, What is Amazon S3?
IAM permissions
When you perform bulk operations you need to create an IAM policy with permissions to allow for the exchange of Amazon resources between Amazon S3, Amazon IoT TwinMaker, Amazon IoT SiteWise, and your local machine. For more information on creating IAM policies, see Creating IAM policies.
The policy statements for Amazon IoT TwinMaker, Amazon IoT SiteWise and Amazon S3 are listed here:
Amazon IoT TwinMaker policy:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetBucketLocation", "s3:ListBucket", "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "iottwinmaker:GetWorkspace", "iottwinmaker:CreateEntity", "iottwinmaker:GetEntity", "iottwinmaker:UpdateEntity", "iottwinmaker:GetComponentType", "iottwinmaker:CreateComponentType", "iottwinmaker:UpdateComponentType", "iottwinmaker:ListEntities", "iottwinmaker:ListComponentTypes", "iottwinmaker:ListTagsForResource", "iottwinmaker:TagResource", "iottwinmaker:UntagResource" ], "Resource": "*" } ] }
Amazon IoT SiteWise policy:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetBucketLocation", "s3:ListBucket", "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "iotsitewise:CreateAsset", "iotsitewise:CreateAssetModel", "iotsitewise:UpdateAsset", "iotsitewise:UpdateAssetModel", "iotsitewise:UpdateAssetProperty", "iotsitewise:ListAssets", "iotsitewise:ListAssetModels", "iotsitewise:ListAssetProperties", "iotsitewise:ListAssetModelProperties", "iotsitewise:ListAssociatedAssets", "iotsitewise:DescribeAsset", "iotsitewise:DescribeAssetModel", "iotsitewise:DescribeAssetProperty", "iotsitewise:AssociateAssets", "iotsitewise:DisassociateAssets", "iotsitewise:AssociateTimeSeriesToAssetProperty", "iotsitewise:DisassociateTimeSeriesFromAssetProperty", "iotsitewise:BatchPutAssetPropertyValue", "iotsitewise:BatchGetAssetPropertyValue", "iotsitewise:TagResource", "iotsitewise:UntagResource", "iotsitewise:ListTagsForResource" ], "Resource": "*" } ] }
Amazon S3 policy:
{ "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetBucketLocation", "s3:ListBucket", "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts" ], "Resource": "*" }
Alternatively you can scope your Amazon S3 policy to only access a single Amazon S3 bucket, see the following policy.
Amazon S3 single bucket scoped policy
{ "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetBucketLocation", "s3:ListBucket", "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts" ], "Resource": [ "arn:aws-cn:s3:::
bucket name
", "arn:aws-cn:s3:::bucket name
/*" ] }
Set access control for a metadataTransferJob
To control what kind of jobs a user can access, add the following IAM policy to the role used to call Amazon IoT TwinMaker.
Note
This policy only allows access to Amazon IoT TwinMaker import and export jobs that transfer resources to and from Amazon S3.
{ "Effect": "Allow", "Action": [ "iottwinmaker:*DataTransferJob*" ], "Resource": "*", "Condition": { "StringLikeIfExists": { "iottwinmaker:sourceType": [ "s3", "iottwinmaker" ], "iottwinmaker:destinationType": [ "iottwinmaker", "s3" ] } } }
Run a bulk operation
This section covers how to perform bulk import and export operations.
Import data from Amazon S3 to Amazon IoT TwinMaker
-
Specify the resources you want to transfer using the Amazon IoT TwinMaker metadataTransferJob schema. Create and store your schema file in your Amazon S3 bucket.
For example schemas, see Import metadata templates.
-
Create a request body and save it as a JSON file. The request body specifies the source and destination for the transfer job. Make sure to specify your Amazon S3 bucket as the source and your Amazon IoT TwinMaker workspace as the destination.
The following is an example of a request body:
{ "metadataTransferJobId": "
your-transfer-job-Id
", "sources": [{ "type": "s3", "s3Configuration": { "location": "arn:aws-cn:s3:::amzn-s3-demo-bucket
/your_import_data.json" } }], "destination": { "type": "iottwinmaker", "iotTwinMakerConfiguration": { "workspace": "arn:aws-cn:iottwinmaker:us-east-1:111122223333:workspace/your-worksapce-name
" } } }Record the file name you gave your request body, you will need it in the next step. In this example the request body is named
createMetadataTransferJobImport.json
. -
Run the following CLI command to invoke
CreateMetadataTransferJob
(replace the input-json file name with the name you gave your request body):aws iottwinmaker create-metadata-transfer-job --region us-east-1 \ --cli-input-json file://createMetadataTransferJobImport.json
This creates a metadataTransferJob and begins the process of the transferring your selected resources.
Export data from Amazon IoT TwinMaker to Amazon S3
-
Create a JSON request body with the appropriate filters to choose the resources you want to export. For this example we use:
{ "metadataTransferJobId": "
your-transfer-job-Id
", "sources": [{ "type": "iottwinmaker", "iotTwinMakerConfiguration": { "workspace": "arn:aws-cn:iottwinmaker:us-east-1:111122223333:workspace/your-workspace-name
", "filters": [{ "filterByEntity": { "entityId": "parent" }}, { "filterByEntity": { "entityId": "child" }}, { "filterByComponentType": { "componentTypeId": "component.type.minimal" }} ] } }], "destination": { "type": "s3", "s3Configuration": { "location": "arn:aws-cn:s3:::amzn-s3-demo-bucket
" } } }The
filters
array lets you specify which resources will be exported. In this example we filter byentity
, andcomponentType
.Make sure to specify your Amazon IoT TwinMaker workspace as the source and your Amazon S3 bucket as the destination of the metadata transfer job.
Save your request body and record the file name, you will need it in the next step. For this example, we named our request body
createMetadataTransferJobExport.json
. -
Run the following CLI command to invoke
CreateMetadataTransferJob
(replace the input-json file name with the name you gave your request body):aws iottwinmaker create-metadata-transfer-job --region us-east-1 \ --cli-input-json file://createMetadataTransferJobExport.json
This creates a metadataTransferJob and begins the process of the transferring your selected resources.
To check or update the status of a transfer job, use the following commands:
-
To cancel a job use the CancelMetadataTransferJob API action. When you call CancelMetadataTransferJob, the API only cancels a running metadataTransferJob, and any resources already exported or imported are not affected by this API call.
-
To retrieve information on a specific job use the GetMetadataTransferJob API action.
Or, you can call GetMetadataTransferJob on an existing transfer job with the following CLI command:
aws iottwinmaker get-metadata-transfer-job --job-id
ExistingJobId
If you call GetMetadataTransferJob on a non-existing Amazon IoT TwinMaker import or export job, you get a
ResourceNotFoundException
error in response. -
To list current jobs, use the ListMetadataTransferJobs API action.
Here is a CLI example that calls ListMetadataTransferJobs with Amazon IoT TwinMaker as the destinationType and
s3
as the sourceType:aws iottwinmaker list-metadata-transfer-jobs --destination-type iottwinmaker --source-type s3
Note
You can change the values for the sourceType and destinationType parameters to match your import or export job's source and destination.
For more examples of CLI commands that invoke these API actions, see Amazon IoT TwinMaker metadataTransferJob examples.
If you encounter any errors during the transfer job, see Error handling.
Error handling
After you create and run a transfer job, you can call GetMetadataTransferJob to diagnose any errors that occurred:
aws iottwinmaker get-metadata-transfer-job \ --metadata-transfer-job-id
your_metadata_transfer_job_id
\ --region us-east-1
Once you see the state of the job turn to COMPLETED
, you can verify
the results of the job. GetMetadataTransferJob returns an object called
MetadataTransferJobProgress
which contains the following fields:
-
failedCount: Indicates the number of resources that failed during the transfer process.
-
skippedCount: Indicates the number of resources that were skipped during the transfer process.
-
succeededCount: Indicates the number of resources that succeeded during the transfer process.
-
totalCount: Indicates the total count of resources involved in the transfer process.
Additionally, a reportUrl element is returned which contains a pre-signed URL. If your transfer job has errors you wish to investigate further, then you can download a full error report using this URL.
Import metadata templates
You can import many components, componentTypes, or entities with a single bulk import operation. The examples in this section show how to do this.
Amazon IoT TwinMaker metadataTransferJob examples
Use the following commands to manage your metadata transfers:
-
CreateMetadataTransferJob API action.
CLI command example:
aws iottwinmaker create-metadata-transfer-job --region us-east-1 \ --cli-input-json file://
yourTransferFileName
.json -
To cancel a job use the CancelMetadataTransferJob API action.
CLI command example:
aws iottwinmaker cancel-metadata-transfer-job --region us-east-1 \ --metadata-transfer-job-id
job-to-cancel-id
When you call CancelMetadataTransferJob, it only cancels a specific metadata transfer job, and any resources already exported or imported are not affected.
-
To retrieve information on a specific job use the GetMetadataTransferJob API action.
CLI command example:
aws iottwinmaker get-metadata-transfer-job \ --metadata-transfer-job-id
your_metadata_transfer_job_id
\ --region us-east-1 \ -
To list current jobs use the ListMetadataTransferJobs API action.
You can filter the results returned by ListMetadataTransferJobs using a JSON file. See the following procedure using the CLI:
-
Create a CLI input JSON file to specify the filters you want to use:
{ "sourceType": "s3", "destinationType": "iottwinmaker", "filters": [{ "workspaceId": "workspaceforbulkimport" }, { "state": "COMPLETED" }] }
Save it and record the file name, you will need it when entering the CLI command.
-
Use the JSON file as an argument to the following CLI command:
aws iottwinmaker list-metadata-transfer-job --region us-east-1 \ --cli-input-json file://ListMetadataTransferJobsExample.json
-