Creating Amazon DataSync locations with the Amazon CLI - Amazon DataSync
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Creating Amazon DataSync locations with the Amazon CLI

Each Amazon DataSync task is made up of a pair of locations in a transfer. The source location defines the storage system or service that you want to read data from. The destination location defines the storage system or service that you want to write data to.

With the Amazon Command Line Interface (Amazon CLI), you can create locations for the following storage systems and services:

  • Network File System (NFS)

  • Server Message Block (SMB)

  • Hadoop Distributed File System (HDFS)

  • Self-managed object storage source locations

  • Amazon Elastic File System (Amazon EFS)

  • Amazon FSx for Windows File Server

  • Amazon FSx for Lustre

  • Amazon FSx for OpenZFS

  • Amazon FSx for NetApp ONTAP

  • Amazon Simple Storage Service (Amazon S3)

For more information, see Where can I transfer my data with Amazon DataSync?.

Creating an NFS location

An NFS location defines a file system on an NFS server that can be read from or written to. You can also create an NFS location by using` the Amazon Web Services Management Console. For more information, see Configuring Amazon DataSync transfers with an NFS file server.

Note

If you are using an NFS location on an Amazon Snowcone device, see Configuring Amazon DataSync transfers with Amazon Snowcone for more information about transferring data to or from that device.

To create an NFS location by using the CLI
  • Use the following command to create an NFS source location.

    $ aws datasync create-location-nfs \ --server-hostname nfs-server-address \ --on-prem-config AgentArns=datasync-agent-arns \ --subdirectory nfs-export-path

    For more information on creating the location, see Accessing NFS file servers.

    DataSync automatically chooses the NFS version that it uses to read from an NFS location. To specify an NFS version, use the optional Version parameter in the NfsMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the NFS location, similar to the ARN shown following.

{ "LocationArn": "arn:aws-cn:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

To make sure that the directory can be mounted, you can connect to any computer that has the same network configuration as your agent and run the following command.

mount -t nfs -o nfsvers=<nfs-server-version <nfs-server-address:<nfs-export-path <test-folder

The following is an example of the command.

mount -t nfs -o nfsvers=3 198.51.100.123:/path_for_sync_to_read_from /temp_folder_to_test_mount_on_local_machine

Creating an SMB location

An SMB location defines a file system on an SMB server that can be read from or written to. You can also create an SMB location by using the console. For more information, see Configuring Amazon DataSync transfers with an SMB file server.

To create an SMB location by using the CLI
  • Use the following command to create an SMB source location.

    aws datasync create-location-smb \ --server-hostname smb-server-address \ --user user-who-can-mount-share \ --domain windows-domain-of-smb-server \ --password user-password \ --agent-arns datasync-agent-arns \ --subdirectory smb-export-path

    The smb-export-path that you provide for the --subdirectory parameter should be a path that's exported by the SMB server. Specify the path by using forward slashes; for example, /path/to/folder. Other SMB clients in your network should be able to access this path.

    DataSync automatically chooses the SMB version that it uses to read from an SMB location. To specify an SMB version, use the optional Version parameter in the SmbMountOptions API operation.

This command returns the Amazon Resource Name (ARN) of the SMB location, similar to the ARN shown following.

{ "LocationArn": "arn:aws-cn:datasync:us-east-1:111222333444:location/loc-0f01451b140b2af49" }

Creating an HDFS location

An HDFS location defines a file system on a Hadoop cluster that can be read from or written to. You can also create an HDFS location by using the Amazon Web Services Management Console. For more information, see Configuring Amazon DataSync transfers with HDFS.

To create an HDFS location by using the Amazon CLI
  • Use the following command to create an HDFS location. In the following example, replace each user input placeholder with your own information.

    aws datasync create-location-hdfs --name-nodes [{"Hostname":"host1", "Port": 8020}] \ --authentication-type "SIMPLE|KERBEROS" \ --agent-arns [arn:aws-cn:datasync:us-east-1:123456789012:agent/agent-01234567890example] \ --subdirectory "/path/to/my/data"

    The following parameters are required in the create-location-hdfs command:

    • name-nodes – Specifies the hostname or IP address of the NameNode in the Hadoop cluster and the TCP port that the NameNode is listening on.

    • authentication-type – The type of authentication to use when connecting to the Hadoop cluster. Specify SIMPLE or KERBEROS.

      If you use SIMPLE authentication, use the --simple-user parameter to specify the user name of the user. If you use KERBEROS authentication, use the --kerberos-principal, --kerberos-keytab, and --kerberos-krb5-conf parameters. For more information, see create-location-hdfs.

    • agent-arns – The ARNs of the DataSync agents to use for the HDFS location.

    The preceding the command returns the location ARN, similar to the following:

    { "arn:aws-cn:datasync:us-east-1:123456789012:location/loc-01234567890example" }

Creating an object storage location

A location is a DataSync endpoint that represents an object storage system hosted on-premises.

For more information about object storage locations, including compatibility requirements, see Configuring Amazon DataSync transfers with an object storage system.

To create an object storage location by using the Amazon CLI
  1. Copy the following create-location-object-storage command:

    aws datasync create-location-object-storage \ --server-hostname object-storage-server.example.com \ --bucket-name your-bucket \ --agent-arns arn:aws-cn:datasync:us-east-1:123456789012:agent/agent-01234567890deadfb
  2. Specify the following required parameters in the command:

    • --server-hostname – Specify the domain name or IP address of your object storage server.

    • --bucket-name – Specify the name of the bucket on your object storage server that you're transferring to or from.

    • --agent-arns – Specify the DataSync agents that you want to connect to your object storage server.

  3. (Optional) Add any of the following parameters to the command:

    • --server-port – Specifies the port that your object storage server accepts inbound network traffic on (for example, port 443).

    • --server-protocol – Specifies the protocol (HTTP or HTTPS) which your object storage server uses to communicate.

    • --access-key – Specifies the access key (for example, a user name) if credentials are required to authenticate with the object storage server.

    • --secret-key – Specifies the secret key (for example, a password) if credentials are required to authenticate with the object storage server.

    • --server-certificate – Specifies a private or self-signed certificate that the DataSync agent will trust when connecting to your object storage server.

      The custom certificate file must have a .pem extension (for example, file:///home/user/mycert.pem).

    • --subdirectory – Specifies the object prefix for your object storage server.

      DataSync only copies objects with this prefix.

    • --tags – Specifies the key-value pair that represents a tag that you want to add to the location resource.

      Tags can help you manage, filter, and search for your resources. We recommend creating a name tag for your location.

  4. Run the create-location-object-storage command.

    You get a response that shows you the location ARN that you just created.

    { "LocationArn": "arn:aws-cn:datasync:us-east-1:123456789012:location/loc-01234567890abcdef" }

Creating an Amazon EFS location

A location is the endpoint for an Amazon EFS file system that can be read from or written to. You can also create this kind of location by using the console. For more information, see Configuring Amazon DataSync transfers with Amazon EFS.

To create an Amazon EFS location by using the Amazon CLI
  1. If you don't have an Amazon EFS file system, create one. For information about how to create an EFS file system, see Getting started with Amazon Elastic File System in the Amazon Elastic File System User Guide.

  2. Identify a subnet that has at least one mount target for that file system. You can see all the mount targets and the subnets associated with an EFS file system by using the describe-mount-targets command.

    aws efs describe-mount-targets \ --region aws-region \ --file-system-id file-system-id
    Note

    The Amazon Web Services Region that you specify is the one where your target S3 bucket or EFS file system is located.

    This command returns information about the target similar to the information shown following.

    { "MountTargets": [ { "OwnerId": "111222333444", "MountTargetId": "fsmt-22334a10", "FileSystemId": "fs-123456ab", "SubnetId": "subnet-f12a0e34", "LifeCycleState": "available", "IpAddress": "11.222.0.123", "NetworkInterfaceId": "eni-1234a044" } ] }
  3. Specify an Amazon EC2 security group that can access the mount target. You can run the following command to find out the security group of the mount target.

    aws efs describe-mount-target-security-groups \ --region aws-region \ --mount-target-id mount-target-id

    The security group that you provide must be able to communicate with the security group on the mount target in the subnet specified.

    The relationship between security group M on the mount target and security group S, which you provide for DataSync to use at this stage, is as follows:

    • Security group M, which you associate with the mount target, must allow inbound access for the TCP protocol on the NFS port (2049) from security group S.

      You can enable an inbound connection either by its IP address (CIDR range) or its security group.

    • Security group S, which you provide to DataSync to access Amazon EFS, should have a rule that enables outbound connections to the NFS port. It enables outbound connections on one of the file system's mount targets.

      You can enable outbound connections either by IP address (CIDR range) or security group.

      For information about security groups and mount targets, see Security groups for Amazon EC2 instances and mount targets in the Amazon Elastic File System User Guide.

  4. Create the location. To create the EFS location, you need the ARNs for your Amazon EC2 subnet, Amazon EC2 security group, and a file system. Because the DataSync API accepts fully qualified ARNs, you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the Amazon Web Services General Reference.

    Use the following command to create an EFS location.

    aws datasync create-location-efs \ --subdirectory /path/to/your/subdirectory \ --efs-filesystem-arn 'arn:aws-cn:elasticfilesystem:region:account-id:file-system/filesystem-id' \ --ec2-config SecurityGroupArns='arn:aws-cn:ec2:region:account-id:security-group/security-group-id',SubnetArn='arn:aws-cn:ec2:region:account-id:subnet/subnet-id'
Note

The Amazon Web Services Region that you specify is the one where your target S3 bucket or EFS file system is located.

The command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws-cn:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Creating an Amazon FSx for Windows File Server location

A location is an endpoint for an FSx for Windows File Server that you can read from or write to.

You can also create an FSx for Windows File Server location by using the console. For more information, see Configuring Amazon DataSync transfers with Amazon FSx for Windows File Server.

To create an FSx for Windows File Server location by using the Amazon CLI
  • Use the following command to create an Amazon FSx location.

    aws datasync create-location-fsx-windows \ --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system/filesystem-id \ --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \ --user smb-user --password password

    In the create-location-fsx-windows command, specify the following:

    • fsx-filesystem-arn – The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

      The DataSync API accepts fully qualified ARNs, and you can construct these ARNs. For information about how to construct ARNs for different services, see Amazon Resource Names (ARNs) in the Amazon Web Services General Reference.

    • security-group-arns – The ARN of an Amazon EC2 security group that can be applied to the network interfaces of the file system's preferred subnet.

    • The Amazon Web Services Region – The Region that you specify is the one where your target Amazon FSx file system is located.

The preceding command returns a location ARN similar to the one shown following.

{ "LocationArn": "arn:aws-cn:datasync:us-west-2:111222333444:location/loc-07db7abfc326c50fb" }

Creating an Amazon FSx for Lustre location

A location is an endpoint for an FSx for Lustre file system that you can read or write to.

You can also create an FSx for Lustre location by using the console. For more information, see Configuring Amazon DataSync transfers with Amazon FSx for Lustre.

To create an FSx for Lustre location by using the Amazon CLI
  • Use the following command to create an FSx for Lustre location.

    aws datasync create-location-fsx-lustre \ --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system:filesystem-id \ --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id

    The following parameters are required in the create-location-fsx-lustre command.

    • fsx-filesystem-arn – The fully qualified Amazon Resource Name (ARN) of the file system that you want to read from or write to.

    • security-group-arns – The ARN of an Amazon EC2 security group to apply to the network interfaces of the file system's preferred subnet.

The preceding command returns a location ARN similar to the following.

{ "LocationArn": "arn:aws-cn:datasync:us-west-2:111222333444:location/loc-07sb7abfc326c50fb" }

Creating an Amazon FSx for OpenZFS location

A location is an endpoint for an FSx for OpenZFS file system that DataSync can access for a transfer. You also can create an FSx for OpenZFS location in the console.

To create an FSx for OpenZFS location by using the Amazon CLI
  1. Copy the following command:

    $ aws datasync create-location-fsx-openzfs \ --fsx-filesystem-arn arn:aws-cn:fsx:region:account-id:file-system/filesystem-id \ --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \ --protocol NFS={}
  2. Specify the following required options in the command:

    • For fsx-filesystem-arn, specify the location file system's fully qualified Amazon Resource Name (ARN). This includes the Amazon Web Services Region where your file system resides, your Amazon Web Services account, and the file system ID.

    • For security-group-arns, specify the ARN of the Amazon EC2 security group that provides access to the network interfaces of your FSx for OpenZFS file system's preferred subnet. This includes the Amazon Web Services Region where your Amazon EC2 instance resides, your Amazon Web Services account, and the security group ID.

      For more information about security groups, see File System Access Control with Amazon VPC in the Amazon FSx for OpenZFS User Guide.

    • For protocol, specify the protocol that DataSync uses to access your file system. (DataSync currently supports only NFS.)

  3. Run the command. You get a response showing the location that you just created.

    { "LocationArn": "arn:aws-cn:datasync:us-west-2:123456789012:location/loc-abcdef01234567890" }

Creating an Amazon FSx for NetApp ONTAP location

A location is an endpoint for an FSx for ONTAP file system that DataSync can access for a transfer. You also can create an FSx for ONTAP location in the console.

To create an FSx for ONTAP location by using the Amazon CLI
  1. Copy the following command:

    $ aws datasync create-location-fsx-ontap \ --storage-virtual-machine-arn arn:aws-cn:fsx:region:account-id:storage-virtual-machine/fs-file-system-id \ --security-group-arns arn:aws-cn:ec2:region:account-id:security-group/group-id \ --protocol data-transfer-protocol={}
  2. Specify the following required options in the command:

    • For storage-virtual-machine-arn, specify the fully qualified Amazon Resource Name (ARN) of a storage virtual machine (SVM) in your file system where you want to copy data to or from.

      This ARN includes the Amazon Web Services Region where your file system resides, your Amazon Web Services account, and the file system and SVM IDs.

    • For security-group-arns, specify the ARNs of the Amazon EC2 security groups that provide access to the network interfaces of your file system's preferred subnet.

      This includes the Amazon Web Services Region where your Amazon EC2 instance resides, your Amazon Web Services account, and your security group IDs. You can specify up to five security group ARNs.

      For more information about security groups, see File System Access Control with Amazon VPC in the Amazon FSx for NetApp ONTAP User Guide.

    • For protocol, configure the protocol that DataSync uses to access your file system's SVM.

      • For NFS, you can use the default configuration:

        --protocol NFS={}

      • For SMB, you must specify a user name and password that can access the SVM:

        --protocol SMB={User=smb-user,Password=smb-password}

  3. Run the command.

    You get a response that shows the location that you just created.

    { "LocationArn": "arn:aws-cn:datasync:us-west-2:123456789012:location/loc-abcdef01234567890" }

Creating an Amazon S3 location

An Amazon S3 location requires an S3 bucket that can be read from or written to.

Important

Before you create your location, make sure that you read the following sections:

DataSync needs an Amazon Identity and Access Management (IAM) role that has the permissions to access your S3 bucket. For more information, see Accessing S3 buckets.

With the following procedure, you create the IAM role, the required IAM policies, and the S3 location by using the Amazon CLI. For DataSync to assume the IAM role, Amazon Security Token Service (Amazon STS) must be activated in your account and Region. For more information about temporary security credentials, see Temporary security credentials in IAM in the IAM User Guide.

To create an S3 location by using the CLI
  1. Create an IAM trust policy that allows DataSync to assume the IAM role required to access your S3 bucket.

    The following is an example of a trust policy.

    { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }
  2. Create a temporary file for the IAM policy, as shown in the following example.

    $ ROLE_FILE=$(mktemp -t sync.iam.role.filename.json) $ IAM_ROLE_NAME='YourBucketAccessRole' $ cat<<EOF> ${ROLE_FILE} { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" }] }
  3. Create an IAM role and attach the IAM policy to it.

    The following command creates an IAM role and attaches the policy to it.

    $ aws iam create-role --role-name ${IAM_ROLE_NAME} --assume-role-policy-document file://${ROLE_FILE} { "Role": { "Path": "/", "RoleName": "YourBucketAccessRole", "RoleId": "role-id", "Arn": "arn:aws-cn:iam::account-id:role/YourBucketAccessRole", "CreateDate": "2018-07-27T02:49:23.117Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": { "Service": "datasync.amazonaws.com" }, "Action": "sts:AssumeRole" }] } } }
  4. Allow the IAM role that you created to write to your S3 bucket.

    Attach to the IAM role an IAM policy that has sufficient permissions to access your S3 bucket. The following example shows the minimum permissions needed for DataSync to read and write to an S3 bucket in an Amazon Web Services Region.

    { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketLocation", "s3:ListBucket", "s3:ListBucketMultipartUploads" ], "Effect": "Allow", "Resource": "YourS3BucketArn" }, { "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListMultipartUploadParts", "s3:PutObjectTagging", "s3:GetObjectTagging", "s3:PutObject" ], "Effect": "Allow", "Resource": "YourS3BucketArn/*" } ] }

    To attach the policy to your IAM role, run the following command.

    $ aws iam attach-role-policy \ --role-name role-name \ --policy-arn 'arn:aws-cn:iam::aws:policy/YourPolicyName'

    For Amazon S3 buckets on Amazon Outposts, use the following policy.

    { "Version": "2012-10-17", "Statement": [{ "Action": [ "s3-outposts:ListBucket", "s3-outposts:ListBucketMultipartUploads" ], "Effect": "Allow", "Resource": [ "s3OutpostsBucketArn", "s3OutpostsAccessPointArn" ], "Condition": { "StringLike": { "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn" } } }, { "Action": [ "s3-outposts:AbortMultipartUpload", "s3-outposts:DeleteObject", "s3-outposts:GetObject", "s3-outposts:ListMultipartUploadParts", "s3-outposts:PutObjectTagging", "s3-outposts:GetObjectTagging", "s3-outposts:PutObject" ], "Effect": "Allow", "Resource": [ "s3OutpostsBucketArn/*", "s3OutpostsAccessPointArn" ], "Condition": { "StringLike": { "s3-outposts:DataAccessPointArn": "s3OutpostsAccessPointArn" } } }, { "Effect": "Allow", "Action": [ "s3-outposts:GetAccessPoint" ], "Resource": "s3OutpostsAccessPointArn" } ] }
  5. Create the location by doing the following:

    1. Copy the following create-location-s3 command:

      $ aws datasync create-location-s3 \ --s3-bucket-arn 'arn:aws-cn:s3:::DOC-EXAMPLE-BUCKET' \ --s3-storage-class 'your-S3-storage-class' \ --s3-config 'BucketAccessRoleArn=arn:aws-cn:iam::account-id:role/role-allowing-DS-operations' \ --subdirectory /your-folder
    2. For --s3-bucket-arn, specify the ARN of the S3 bucket that you want to use as a location.

      (When creating your DataSync task later, you specify whether this location is a source or destination location.)

      If your S3 bucket is located on an Amazon Outposts resource, you must specify an Amazon S3 access point. For more information, see Managing data access with Amazon S3 access points in the Amazon S3 User Guide.

    3. For --s3-storage-class, specify a storage class that you want your objects to use when Amazon S3 is a transfer destination.

      For more information, see Storage class considerations with Amazon S3 transfers. DataSync by default uses the S3 Outposts storage class for Amazon S3 on Outposts.

    4. For --s3-config, specify the ARN of the IAM role that DataSync needs to access your bucket.

    5. For --subdirectoy, specify a prefix in the S3 bucket that DataSync reads from or writes to (depending on whether the bucket is a source or destination location).

      Warning

      DataSync can't transfer objects with a prefix that begins with a slash (/) or includes //, /./, or /../ patterns. For example:

      • /photos

      • photos//2006/January

      • photos/./2006/February

      • photos/../2006/March

    6. (Amazon S3 on Outposts only) Add the --agent-arns option to the command and use it to specify the ARN of the DataSync agent on your Outpost.

      For more information, see Deploy your agent on Amazon Outposts.

    7. Run the create-location-s3 command.

      If the command is successful, you get a response that shows you the ARN of the location that you created. For example:

      { "LocationArn": "arn:aws-cn:datasync:us-east-1:111222333444:location/loc-0b3017fc4ba4a2d8d" }