Other use cases for Amazon DataSync
Learn about about some less-common use cases with Amazon DataSync that are not common to most users.
Transferring data in opposite directions
Transferring data in opposite directions allows for workflows where the active application moves between locations. Amazon DataSync doesn't support workflows where multiple active applications write to both locations at the same time. Use the steps in the following procedure to configure DataSync to transfer data in opposite directions.
To configure DataSync to data transfers in opposite directions
Create a location and name it Location A.
Create a second location and name it Location B.
Create a task, name it Task A-B, and then configure Location A as the source location and Location B as the destination location.
Create a second task, name it Task B-A, and then configure Location B as the source location and Location A as the destination location.
To update Location B with data from Location A, run Task A-B.
To update Location A with data from Location B, run Task B-A.
Don't run these two tasks concurrently. DataSync can transfer files in opposite directions periodically. However, it doesn't support workflows where multiple active applications write to both Location A and Location B simultaneously.
Using multiple DataSync tasks to transfer to the same S3 bucket
In certain use cases, you might want different tasks to write to the same Amazon S3 bucket. In this case, you create different folders in the S3 bucket for each of the task. This approach prevents file name conflicts between the tasks, and also means that you can set different permissions for each of folders.
For example, you might have three tasks: task1
, task2
, and
task3
write to an S3 bucket named MyBucket
.
You create three folders in the bucket:
s3://MyBucket/task1
s3://MyBucket/task2
s3://MyBucket/task3
For each task, you choose the folder in MyBucket
that corresponds to the
task as the destination, and set different permissions for each of the three folders.
Allowing DataSync to access a restricted S3 bucket
In some cases, you might want to limit access to your Amazon S3 bucket. You can edit the S3 bucket policy so that DataSync can still access the bucket when you run a task.
To allow DataSync to access a restricted S3 bucket
-
Copy the following sample policy.
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Principal": "*", "Action": "s3:*", "Resource": [ "arn:aws-cn:s3:::
bucket-name
", "arn:aws-cn:s3:::bucket-name
/*" ], "Condition": { "StringNotLike": { "aws:userid": [ "datasync-role-id
:*", "your-role-id
" ] } } } ] } -
In the sample policy, replace these values:
-
bucket-name
: The name of the S3 bucket that you're restricting access to. -
datasync-role-id
: The ID of the IAM role that DataSync accesses the S3 bucket with. Run the following Amazon CLI command to get the IAM role ID:aws iam get-role --role-name
datasync-iam-role-name
In the output, look for the
RoleId
value:"RoleId": "ANPAJ2UCCR6DPCEXAMPLE"
-
your-role-id
: The ID of the IAM role that you create the DataSync location for the S3 bucket with. Run the following command to get the IAM role ID:aws iam get-role --role-name
your-iam-role-name
In the output, look for the
UserId
value:"RoleId": "AIDACKCEVSQ6C2EXAMPLE"
-
-
Add this policy to your S3 bucket policy. For more information, see how to edit a bucket policy in the Amazon S3 User Guide.
Once you've updated the S3 bucket policy, you must add additional IAM roles or users to the policy for those who need to access the S3 bucket.