Creating Amazon DataSync locations with the Amazon CLI

To create an NFS location by using the CLI

• Use the following command to create an NFS source location.

  $ aws datasync create-location-nfs \
      --server-hostname nfs-server-address \
      --on-prem-config AgentArns=datasync-agent-arns \
      --subdirectory nfs-export-path

  For more information on creating the location, see Accessing NFS file servers.

  DataSync automatically chooses the NFS version that it uses to read from an NFS location. To specify an NFS version, use the optional Version parameter in the NfsMountOptions API operation.

To create an SMB location by using the CLI

• Use the following command to create an SMB source location.

  aws datasync create-location-smb \
      --server-hostname smb-server-address \
      --user user-who-can-mount-share \
      --domain windows-domain-of-smb-server \
      --password user-password \
      --agent-arns datasync-agent-arns \
      --subdirectory smb-export-path

  The smb-export-path that you provide for the --subdirectory parameter should be a path that's exported by the SMB server. Specify the path by using forward slashes; for example, /path/to/folder. Other SMB clients in your network should be able to access this path.

  DataSync automatically chooses the SMB version that it uses to read from an SMB location. To specify an SMB version, use the optional Version parameter in the SmbMountOptions API operation.

To create an HDFS location by using the Amazon CLI

• Use the following command to create an HDFS location. In the following example, replace each user input placeholder with your own information.

  aws datasync create-location-hdfs --name-nodes [{"Hostname":"host1", "Port": 8020}] \
      --authentication-type "SIMPLE|KERBEROS" \
      --agent-arns [arn:aws-cn:datasync:us-east-1:123456789012:agent/agent-01234567890example] \
      --subdirectory "/path/to/my/data"

  The following parameters are required in the create-location-hdfs command:

  • name-nodes – Specifies the hostname or IP address of the NameNode in the Hadoop cluster and the TCP port that the NameNode is listening on.

  • authentication-type – The type of authentication to use when connecting to the Hadoop cluster. Specify SIMPLE or KERBEROS.

    If you use SIMPLE authentication, use the --simple-user parameter to specify the user name of the user. If you use KERBEROS authentication, use the --kerberos-principal, --kerberos-keytab, and --kerberos-krb5-conf parameters. For more information, see create-location-hdfs.

  • agent-arns – The ARNs of the DataSync agents to use for the HDFS location.

  The preceding the command returns the location ARN, similar to the following:

  {
      "arn:aws-cn:datasync:us-east-1:123456789012:location/loc-01234567890example"
  }

To create an object storage location by using the Amazon CLI

1. Copy the following create-location-object-storage command:

   aws datasync create-location-object-storage \
       --server-hostname object-storage-server.example.com \
       --bucket-name your-bucket \
       --agent-arns arn:aws-cn:datasync:us-east-1:123456789012:agent/agent-01234567890deadfb

2. Specify the following required parameters in the command:

   • --server-hostname – Specify the domain name or IP address of your object storage server.

   • --bucket-name – Specify the name of the bucket on your object storage server that you're transferring to or from.

   • --agent-arns – Specify the DataSync agents that you want to connect to your object storage server.

3. (Optional) Add any of the following parameters to the command:

   • --server-port – Specifies the port that your object storage server accepts inbound network traffic on (for example, port 443).

   • --server-protocol – Specifies the protocol (HTTP or HTTPS) which your object storage server uses to communicate.

   • --access-key – Specifies the access key (for example, a user name) if credentials are required to authenticate with the object storage server.

   • --secret-key – Specifies the secret key (for example, a password) if credentials are required to authenticate with the object storage server.

   • --server-certificate – Specifies a private or self-signed certificate that the DataSync agent will trust when connecting to your object storage server.

     The custom certificate file must have a .pem extension (for example, file:///home/user/mycert.pem).

   • --subdirectory – Specifies the object prefix for your object storage server.

     DataSync only copies objects with this prefix.

   • --tags – Specifies the key-value pair that represents a tag that you want to add to the location resource.

     Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.

4. Run the create-location-object-storage command.

   You get a response that shows you the location ARN that you just created.

   {
       "LocationArn": "arn:aws-cn:datasync:us-east-1:123456789012:location/loc-01234567890abcdef"
   }

To create an Amazon EFS location by using the Amazon CLI

1. If you don't have an Amazon EFS file system, create one. For information about how to create an EFS file system, see Getting started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

2. Identify a subnet that has at least one mount target for that file system. You can see all the mount targets and the subnets associated with an EFS file system by using the describe-mount-targets command.

   aws efs describe-mount-targets \
       --region aws-region  \
       --file-system-id file-system-id

   Note

   The Amazon Web Services Region that you specify is the one where your target S3 bucket or EFS file system is located.

   This command returns information about the target similar to the information shown following.

   {
       "MountTargets": [
           {
               "OwnerId": "111222333444",
               "MountTargetId": "fsmt-22334a10",
               "FileSystemId": "fs-123456ab",
               "SubnetId": "subnet-f12a0e34",
               "LifeCycleState": "available",
               "IpAddress": "11.222.0.123",
               "NetworkInterfaceId": "eni-1234a044"
           }
       ]
   }
   

3. Specify an Amazon EC2 security group that can access the mount target. You can run the following command to find out the security group of the mount target.

   aws efs describe-mount-target-security-groups \
       --region aws-region \
       --mount-target-id mount-target-id

   The security group that you provide must be able to communicate with the security group on the mount target in the subnet specified.

   The relationship between security group M on the mount target and security group S, which you provide for DataSync to use at this stage, is as follows:

   • Security group M, which you associate with the mount target, must allow inbound access for the TCP protocol on the NFS port (2049) from security group S.

     You can enable an inbound connection either by its IP address (CIDR range) or its security group.

   • Security group S, which you provide to DataSync to access Amazon EFS, should have a rule that enables outbound connections to the NFS port. It enables outbound connections on one of the file system's mount targets.

     You can enable outbound connections either by IP address (CIDR range) or security group.

     For information about security groups and mount targets, see Security groups for Amazon EC2 instances and mount targets in the Amazon Elastic File System User Guide.

4. Create the location. To create the EFS location, you need the ARNs for your Amazon EC2 subnet, Amazon EC2 security group, and a file system. Because the DataSync API accepts fully qualified ARNs, you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the Amazon Web Services General Reference.

   Use the following command to create an EFS location.

   aws datasync create-location-efs \
       --subdirectory /path/to/your/subdirectory \
       --efs-filesystem-arn 'arn:aws-cn:elasticfilesystem:region:account-id:file-system/filesystem-id' \
       --ec2-config SecurityGroupArns='arn:aws-cn:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws-cn:ec2:region:account-id:subnet/subnet-id'

To create an FSx for Windows File Server location by using the Amazon CLI

• Use the following command to create an Amazon FSx location.

  aws datasync create-location-fsx-windows \
      --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system/filesystem-id \
      --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \
      --user smb-user --password password

  In the create-location-fsx-windows command, specify the following:

  • fsx-filesystem-arn – The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

    The DataSync API accepts fully qualified ARNs, and you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the Amazon Web Services General Reference.

  • security-group-arns – The ARN of an Amazon EC2 security group that can be applied to the network interfaces of the file system's preferred subnet.

  • The Amazon Web Services Region – The Region that you specify is the one where your target Amazon FSx file system is located.

To create an FSx for Lustre location by using the Amazon CLI

• Use the following command to create an FSx for Lustre location.

  aws datasync create-location-fsx-lustre \
      --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system:filesystem-id \
      --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id

  The following parameters are required in the create-location-fsx-lustre command.

  • fsx-filesystem-arn – The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

  • security-group-arns – The ARN of an Amazon EC2 security group to apply to the network interfaces of the file system's preferred subnet.

To create an FSx for OpenZFS location by using the Amazon CLI

1. Copy the following command:

   $ aws datasync create-location-fsx-openzfs \
      --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system/filesystem-id \
      --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \
      --protocol NFS={}

2. Specify the following required options in the command:

   • For fsx-filesystem-arn, specify the location file system's fully qualified Amazon Resource Name (ARN). This includes the Amazon Web Services Region where your file system resides, your Amazon Web Services account, and the file system ID.

   • For security-group-arns, specify the ARN of the Amazon EC2 security group that provides access to the network interfaces of your FSx for OpenZFS file system's preferred subnet. This includes the Amazon Web Services Region where your Amazon EC2 instance resides, your Amazon Web Services account, and the security group ID.

     For more information about security groups, see File System Access Control with Amazon VPC in the Amazon FSx for OpenZFS User Guide.

   • For protocol, specify the protocol that DataSync uses to access your file system. (DataSync currently supports only NFS.)

3. Run the command. You get a response showing the location that you just created.

   { 
       "LocationArn": "arn:aws-cn:datasync:us-west-2:123456789012:location/loc-abcdef01234567890" 
   }

To create an FSx for ONTAP location by using the Amazon CLI

1. Copy the following command:

   $ aws datasync create-location-fsx-ontap \
      --storage-virtual-machine-arn arn:aws-cn:fsx:region:account-id:storage-virtual-machine/fs-file-system-id \
      --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \
      --protocol data-transfer-protocol={}

2. Specify the following required options in the command:

   • For storage-virtual-machine-arn, specify the fully qualified Amazon Resource Name (ARN) of a storage virtual machine (SVM) in your file system where you want to copy data to or from.

     This ARN includes the Amazon Web Services Region where your file system resides, your Amazon Web Services account, and the file system and SVM IDs.

   • For security-group-arns, specify the ARNs of the Amazon EC2 security groups that provide access to the network interfaces of your file system's preferred subnet.

     This includes the Amazon Web Services Region where your Amazon EC2 instance resides, your Amazon Web Services account, and your security group IDs. You can specify up to five security group ARNs.

     For more information about security groups, see File System Access Control with Amazon VPC in the Amazon FSx for NetApp ONTAP User Guide.

   • For protocol, configure the protocol that DataSync uses to access your file system's SVM.

     • For NFS, you can use the default configuration:

       --protocol NFS={}

     • For SMB, you must specify a user name and password that can access the SVM:

       --protocol SMB={User=smb-user,Password=smb-password}

3. Run the command.

   You get a response that shows the location that you just created.

   { 
       "LocationArn": "arn:aws-cn:datasync:us-west-2:123456789012:location/loc-abcdef01234567890" 
   }

To create an S3 location by using the CLI

1. Create an IAM trust policy that allows DataSync to assume the IAM role required to access your S3 bucket.

   The following is an example of a trust policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "datasync.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }
   

2. Create a temporary file for the IAM policy, as shown in the following example.

   $ ROLE_FILE=$(mktemp -t sync.iam.role.filename.json)
   $ IAM_ROLE_NAME='YourBucketAccessRole'
   
   $ cat<<EOF> ${ROLE_FILE}
   {
       "Version": "2012-10-17",
       "Statement": [{
           "Effect": "Allow",
           "Principal": {
               "Service": "datasync.amazonaws.com"
           },
           "Action": "sts:AssumeRole"
       }]
   }

3. Create an IAM role and attach the IAM policy to it.

   The following command creates an IAM role and attaches the policy to it.

   $ aws iam create-role --role-name ${IAM_ROLE_NAME} --assume-role-policy-document file://${ROLE_FILE}
   {
       "Role": {
           "Path": "/",
           "RoleName": "YourBucketAccessRole",
           "RoleId": "role-id",
           "Arn": "arn:aws-cn:iam::account-id:role/YourBucketAccessRole",
           "CreateDate": "2018-07-27T02:49:23.117Z",
           "AssumeRolePolicyDocument": {
               "Version": "2012-10-17",
               "Statement": [{
                   "Effect": "Allow",
                   "Principal": {
                       "Service": "datasync.amazonaws.com"
                   },
                   "Action": "sts:AssumeRole"
               }]
           }
       }
   }

4. Allow the IAM role that you created to write to your S3 bucket.

   Attach to the IAM role an IAM policy that has sufficient permissions to access your S3 bucket. The following example shows the minimum permissions needed for DataSync to read and write to an S3 bucket in an Amazon Web Services Region.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "s3:GetBucketLocation",
                   "s3:ListBucket",
                   "s3:ListBucketMultipartUploads"
               ],
               "Effect": "Allow",
               "Resource": "YourS3BucketArn"
           },
           {
               "Action": [
                   "s3:AbortMultipartUpload",
                   "s3:DeleteObject",
                   "s3:GetObject",
                   "s3:ListMultipartUploadParts",
                   "s3:PutObjectTagging",
                   "s3:GetObjectTagging",
                   "s3:PutObject"
               ],
               "Effect": "Allow",
               "Resource": "YourS3BucketArn/*"
           }
       ]
   }

   To attach the policy to your IAM role, run the following command.

   $ aws iam attach-role-policy \
       --role-name role-name  \
       --policy-arn 'arn:aws-cn:iam::aws:policy/YourPolicyName'

   For Amazon S3 buckets on Amazon Outposts, use the following policy.

   {
       "Version": "2012-10-17",
       "Statement": [{
               "Action": [
                   "s3-outposts:ListBucket",
                   "s3-outposts:ListBucketMultipartUploads"
               ],
               "Effect": "Allow",
               "Resource": [
                   "s3OutpostsBucketArn",
                   "s3OutpostsAccessPointArn"
               ],
               "Condition": {
                   "StringLike": {
                       "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn"
                   }
               }
           },
           {
               "Action": [
                   "s3-outposts:AbortMultipartUpload",
                   "s3-outposts:DeleteObject",
                   "s3-outposts:GetObject",
                   "s3-outposts:ListMultipartUploadParts",
                   "s3-outposts:PutObjectTagging",
                   "s3-outposts:GetObjectTagging",
                   "s3-outposts:PutObject"
               ],
               "Effect": "Allow",
               "Resource": [
                   "s3OutpostsBucketArn/*",
                   "s3OutpostsAccessPointArn"
               ],
               "Condition": {
                   "StringLike": {
                       "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn"
                   }
               }
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3-outposts:GetAccessPoint"
               ],
               "Resource": "s3OutpostsAccessPointArn"
           }
       ]
   }

5. Create the S3 location.

   Use the following command to create your Amazon S3 location.

   $ aws datasync create-location-s3 \
       --s3-bucket-arn 'arn:aws-cn:s3:::DOC-EXAMPLE-BUCKET' \
       --s3-storage-class 'your-S3-storage-class' \
       --s3-config 'BucketAccessRoleArn=arn:aws-cn:iam::account-id:role/role-allowing-DS-operations' \
       --subdirectory /your-folder

   The command returns a location ARN similar to the one shown following.

   {
       "LocationArn": "arn:aws-cn:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d"
   }

   The location type information is encoded in the LocationUri. In this example, the s3:// prefix in LocationUri shows the location's type.

   If your Amazon S3 bucket is located on an Amazon Outpost, you must deploy an Amazon EC2 agent on your Outpost. The agent must be in a virtual private cloud (VPC) that's allowed to access the access point specified in the command. The agent also must be activated in the parent Region for the Outpost, and be able to route to the Amazon S3 on Amazon Outposts endpoints for the bucket. For more information about launching a DataSync agent on Amazon Outposts, see Deploy your agent on Amazon Outposts.

   Use the following command to create an Amazon S3 location on your Outpost.

   aws datasync create-location-s3 \
       --s3-bucket-arn access-point-arn \
       --s3-config BucketAccessRoleArn=arn:aws-cn:iam::account-id:role/role-allowing-DS-operations \
       --agent-arns arn-of-datasync-agent-in-vpc-that-can-access-your-s3-access-point