Configuring Amazon DataSync transfers with Amazon S3

1. Open the IAM console at https://console.amazonaws.cn/iam/.

2. In the left navigation pane, under Access management, choose Roles, and then choose Create role.

3. On the Select trusted entity page, for Trusted entity type, choose Amazon Web Service.

4. For Use case, choose DataSync in the dropdown list and select DataSync. Choose Next.

5. On the Add permissions page, choose Next. Give your role a name and choose Create role.

6. On the Roles page, search for the role that you just created and choose its name.

7. On the role's details page, choose the Permissions tab. Choose Add permissions then Create inline policy.

8. Choose the JSON tab and add the permissions required to access your bucket into the policy editor.

9. Choose Next. Give your policy a name and choose Create policy.

10. (Recommended) To prevent the cross-service confused deputy problem, do the following:

    1. On the role's details page, choose the Trust relationships tab. Choose Edit trust policy.

    2. Update the trust policy by using the following example, which includes the aws:SourceArn and aws:SourceAccount global condition context keys:

       {
           "Version": "2012-10-17",
           "Statement": [{
               "Effect": "Allow",
               "Principal": {
                   "Service": "datasync.amazonaws.com"
               },
               "Action": "sts:AssumeRole",
               "Condition": {
                   "StringEquals": {
                       "aws:SourceAccount": "account-id"
                   },
                   "StringLike": {
                       "aws:SourceArn": "arn:aws-cn:datasync:region:account-id:*"
                   }
               }
           }]
       }

    3. Choose Update policy.

1. Copy the following S3 bucket policy.

   {
       "Version": "2012-10-17",
       "Statement": [{
           "Sid": "Deny-access-to-bucket",
           "Effect": "Deny",
           "Principal": "*",
           "Action": "s3:*",
           "Resource": [
               "arn:aws:s3:::bucket-name",
               "arn:aws:s3:::bucket-name/*"
           ],
           "Condition": {
               "StringNotLike": {
                   "aws:userid": [
                       "datasync-iam-role-id:*",
                       "your-iam-role-id"
                   ]
               }
           }
       }]
   }

2. In the policy, replace the following values:

   • bucket-name – Specify the name of the restricted S3 bucket.

   • datasync-iam-role-id – Specify the ID of the IAM role that DataSync uses to access the bucket.

     Run the following Amazon CLI command to get the IAM role ID:

     aws iam get-role --role-name datasync-iam-role-name

     In the output, look for the RoleId value:

     "RoleId": "ANPAJ2UCCR6DPCEXAMPLE"

   • your-iam-role-id – Specify the ID of the IAM role that you use to create your DataSync location for the bucket.

     Run the following command to get the IAM role ID:

     aws iam get-role --role-name your-iam-role-name

     In the output, look for the RoleId value:

     "RoleId": "AIDACKCEVSQ6C2EXAMPLE"

3. Add this policy to your S3 bucket policy.

4. When you're done using DataSync with the restricted bucket, remove the conditions for both IAM roles from the bucket policy.

1. Open the Amazon DataSync console at https://console.amazonaws.cn/datasync/.

2. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

3. For Location type, choose Amazon S3.

4. For S3 bucket, choose the bucket that you want to use as a location.

   If your S3 bucket is located on an Amazon Outposts resource, you must specify an Amazon S3 access point. For more information, see Managing data access with Amazon S3 access points in the Amazon S3 User Guide.

5. For S3 storage class, choose a storage class that you want your objects to use when Amazon S3 is a transfer destination.

   For more information, see Storage class considerations with Amazon S3 transfers.

6. (Amazon S3 on Outposts only) For Agents, specify the Amazon Resource Name (ARN) of the DataSync agent on your Outpost.

   For more information, see Deploying your agent on Amazon Outposts.

7. For Folder, enter a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).

   Warning

   DataSync can't transfer objects with a prefix that begins with a slash (/) or includes //, /./, or /../ patterns. For example:

   • /photos

   • photos//2006/January

   • photos/./2006/February

   • photos/../2006/March

8. For IAM role, do one of the following:

   • Choose Autogenerate for DataSync to automatically create an IAM role with the permissions required to access the S3 bucket.

     If DataSync previously created an IAM role for this S3 bucket, that role is chosen by default.

   • Choose a custom IAM role that you created. For more information, see Creating an IAM role for DataSync to access your Amazon S3 location.

9. (Optional) Choose Add tag to tag your Amazon S3 location.

   Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.

10. Choose Create location.

1. Copy the following create-location-s3 command:

   aws datasync create-location-s3 \
       --s3-bucket-arn 'arn:aws-cn:s3:::DOC-EXAMPLE-BUCKET' \
       --s3-storage-class 'your-S3-storage-class' \
       --s3-config 'BucketAccessRoleArn=arn:aws-cn:iam::account-id:role/role-allowing-DS-operations' \
       --subdirectory /your-folder

2. For --s3-bucket-arn, specify the ARN of the S3 bucket that you want to use as a location.

   (When creating your DataSync task later, you specify whether this location is a source or destination location.)

   If your S3 bucket is located on an Amazon Outposts resource, you must specify an Amazon S3 access point. For more information, see Managing data access with Amazon S3 access points in the Amazon S3 User Guide.

3. For --s3-storage-class, specify a storage class that you want your objects to use when Amazon S3 is a transfer destination.

   For more information, see Storage class considerations with Amazon S3 transfers. DataSync by default uses the S3 Outposts storage class for Amazon S3 on Outposts.

4. For --s3-config, specify the ARN of the IAM role that DataSync needs to access your bucket.

   For more information, see Creating an IAM role for DataSync to access your Amazon S3 location.

5. For --subdirectoy, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).

   Warning

   DataSync can't transfer objects with a prefix that begins with a slash (/) or includes //, /./, or /../ patterns. For example:

   • /photos

   • photos//2006/January

   • photos/./2006/February

   • photos/../2006/March

6. (Amazon S3 on Outposts only) Add the --agent-arns option to the command and use it to specify the ARN of the DataSync agent on your Outpost.

   For more information, see Deploying your agent on Amazon Outposts.

7. Run the create-location-s3 command.

   If the command is successful, you get a response that shows you the ARN of the location that you created. For example:

   {
       "LocationArn": "arn:aws-cn:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d"
   }

To create your DataSync agent

1. Deploy an Amazon EC2 agent in your commercial Region.

2. Configure your agent to use a VPC service endpoint.

3. Activate your agent.

To create your DataSync agent

1. Deploy an Amazon EC2 agent in your Amazon GovCloud (US) Region.

2. Configure your agent to use a VPC service endpoint.

3. Activate your agent.

1. Open the Amazon DataSync console at https://console.amazonaws.cn/datasync/.

2. Make sure that you're in the same Region where you plan to run your task.

3. In the left navigation pane, expand Data transfer, then choose Locations and Create location.

4. For Location type, choose Object storage.

5. For Agents, choose the DataSync agent that you created for this transfer.

6. For Server, enter an Amazon S3 endpoint for your bucket by using one of the following formats:

   • Commercial Region bucket: s3.your-region.amazonaws.com

   • Amazon GovCloud (US) Region bucket: s3.your-gov-region.amazonaws.com

   For a list of Amazon S3 endpoints, see the Amazon Web Services General Reference.

7. For Bucket name, enter the name of the S3 bucket.

8. For Folder, enter a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).

   Warning

   DataSync can't transfer objects with a prefix that begins with a slash (/) or includes //, /./, or /../ patterns. For example:

   • /photos

   • photos//2006/January

   • photos/./2006/February

   • photos/../2006/March

9. Select Requires credentials and do the following:

   • For Access key, enter the access key for an IAM user that can access the bucket.

   • For Secret key, enter the same IAM user’s secret key.

10. (Optional) Choose Add tag to tag your location.

    Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.

11. Choose Create location.

1. Copy the following create-location-object-storage command:

   aws datasync create-location-object-storage \
       --server-hostname s3-endpoint \
       --bucket-name your-bucket \
       --agent-arns arn:aws-cn:datasync:your-region:123456789012:agent/agent-01234567890deadfb

2. For the --server-hostname parameter, specify an Amazon S3 endpoint for your bucket by using one of the following formats:

   • Commercial Region bucket: s3.your-region.amazonaws.com

   • Amazon GovCloud (US) Region bucket: s3.your-gov-region.amazonaws.com

   For the Region in the endpoint, make sure that you specify the same Region where you plan to run your task.

   For a list of Amazon S3 endpoints, see the Amazon Web Services General Reference.

3. For the --bucket-name parameter, specify the name of the S3 bucket.

4. For the --agent-arns parameter, specify the DataSync agent that you created for this transfer.

5. For the --access-key parameter, specify the access key for an IAM user that can access the bucket.

6. For the --secret-key parameter, enter the same IAM user's secret key.

7. (Optional) For the --subdirectory parameter, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).

   Warning

   DataSync can't transfer objects with a prefix that begins with a slash (/) or includes //, /./, or /../ patterns. For example:

   • /photos

   • photos//2006/January

   • photos/./2006/February

   • photos/../2006/March

8. (Optional) For the --tags parameter, specify key-value pairs that represent tags for the location resource.

   Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.

9. Run the create-location-object-storage command.

   You get a response that shows you the location ARN that you just created.

   {
       "LocationArn": "arn:aws-cn:datasync:us-east-1:123456789012:location/loc-01234567890abcdef"
   }