Using Amazon DataSync to Transfer Files - Amazon Snowcone User Guide
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Using Amazon DataSync to Transfer Files

Amazon DataSync is an online data transfer service that simplifies, automates, and accelerates copying large amounts of data to and from Amazon storage services over the internet or Amazon Direct Connect. DataSync agent comes pre-installed on your Snowcone device. It can transfer data between the device and Amazon S3 buckets, Amazon EFS, and Amazon FSx for Windows File Server. Amazon DataSync automatically handles moving files and objects, scheduling data transfers, monitoring the progress of transfers, encryption, verification of data transfers, and notifying customers of any issues.

The DataSync agent is pre-installed on your Snowcone device as an AMI during the Snowcone job preparation. To transfer data online to Amazon, connect the Snowcone device to the external network and use Amazon OpsHub or the CLI to launch the DataSync agent AMI. Activate the DataSync agent the Amazon Management Console or use the CLI, and set up your online data transfer task between the Snowcone NFS store, and Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server.

You can use Amazon DataSync running on Snowcone for the following:

  • Edge computing applications, to collect data, process the data to gain immediate insight, and then transfer the data online to Amazon.

  • Transfer data that is continuously generated by sensors or machines online to Amazon in a factory or at other edge locations.

  • Distribute media, scientific, or other content online from Amazon storage services to your partners and customers.

  • Aggregate content by transferring media, scientific or other content online from your edge locations to Amazon.

  • Ensure you use a static IP address for the VNI for the DataSync agent. Using DHCP may cause issues because any device reboots can mean that you’re using a different IP address, forcing a the need to reconfigure both the DataSync agent and DataSync activation inAmazon cloud.

  • To use Datasync agent on Snowcone, you need to ensure that the “AllowedHosts” entries includes the DataSync Agent IP address in your NFS configuration on the Snowcone. This may require you to stop and restart the NFS service on Snowcone to enable the DataSync agent.

  • Whenever you stop the NFS service, you should also stop the DataSync agent too. Note: If you stop the NFS service which has allow the listed DataSync agent, it will stop Datasync agent from working.

  • Check the maximum number of files per task for a Snowcone HDD device (the # might be higher for Snowcone SSD). If you exceed the 200,000 files limit, the DataSync task will report a memory allocation error and abort execution.

For one-time edge compute or data transfer workflows or for Snowcone workflows in edge locations without a wide area network (WAN) link or inadequate WAN bandwidth, you should ship the Snowcone device back to Amazon to complete the data transfer.