Transferring files using the Amazon S3 adapter for data migration to or from Snowball Edge
Following is an overview of the Amazon S3 adapter, which you can use to transfer data programmatically to and from S3 buckets already on the Amazon Snowball Edge device using Amazon S3 REST API actions. This Amazon S3 REST API support is limited to a subset of actions. You can use this subset of actions with one of the Amazon SDKs to transfer data programmatically. You can also use the subset of supported Amazon Command Line Interface (Amazon CLI) commands for Amazon S3 to transfer data programmatically.
If your solution uses the Amazon SDK for Java version 1.11.0 or newer, you must use the following
S3ClientOptions
:
-
disableChunkedEncoding()
– Indicates that chunked encoding is not supported with the interface. -
setPathStyleAccess(true)
– Configures the interface to use path-style access for all requests.
For more information, see Class S3ClientOptions.Builder in the Amazon AppStream SDK for Java.
Important
We recommend that you use only one method at a time to read and write data to a local bucket on an Amazon Snowball Edge device. Using both the NFS interface and the Amazon S3 adapter on the same bucket at the same time can result in read/write conflicts.
Amazon Snowball Edge quotas details the limits.
For Amazon services to work properly on a Snowball Edge, you must allow the ports for the services. For details, see Port requirements for Amazon services on a Snowball Edge.
Topics
- Downloading and installing the Amazon CLI version 1.16.14 for use with the Amazon S3 adapter
- Using the Amazon CLI and API operations on Snowball Edge devices
- Getting and using local Amazon S3 credentials on Snowball Edge
- Unsupported Amazon S3 features for the Amazon S3 adapter on Snowball Edge
- Batching small files to improve data transfer performance to Snowball Edge
- Supported Amazon CLI commands for data transfer to or from Snowball Edge
- Supported Amazon S3 REST API actions on Snowball Edge for data transfer
Downloading and installing the Amazon CLI version 1.16.14 for use with the Amazon S3 adapter
Currently, Snowball Edge devices support only version 1.16.14 and earlier of the Amazon CLI for use with the Amazon S3 adapter. Newer versions of the Amazon CLI are not compatible with the Amazon S3 adapter because they do not support all of the functionality of the S3 adapter.
Note
If you are using Amazon S3 compatible storage on Snowball Edge, you can use the latest version of the Amazon CLI. To download and use the latest version, see Amazon Command Line Interface User Guide.
Install the Amazon CLI on Linux operating systems
Run this chained command:
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle-1.16.14.zip" -o "awscli-bundle.zip";unzip awscli-bundle.zip;sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws;/usr/local/bin/aws --version;
Install the Amazon CLI on Windows operating systems
Download and run the installer file for your operating system:
Using the Amazon CLI and API operations on Snowball Edge devices
When using the Amazon CLI or API operations to issue IAM, Amazon S3, and Amazon EC2 commands on
Snowball Edge, you must specify the Region as "snow
." You can do this
using Amazon configure
or within the command itself, as in the
following examples.
aws configure --profile abc Amazon Access Key ID [None]: AKIAIOSFODNN7EXAMPLE Amazon Secret Access Key [None]: 1234567 Default region name [None]: snow Default output format [None]: json
Or
aws s3 ls --profile snowballEdge --endpoint http://192.0.2.0:8080 --region snow
Authorization with the Amazon S3 API interface for Amazon Snowball
When you use the Amazon S3 adapter, every interaction is signed with the Amazon Signature Version 4 algorithm by default. This authorization is used only to verify the data traveling from its source to the interface. All encryption and decryption happens on the device. Unencrypted data is never stored on the device.
When using the interface, keep the following in mind:
-
To get the local Amazon S3 credentials to sign your requests to the Amazon Snowball Edge device, run the
snowballEdge list-access-keys
andsnowballEdge get-secret-access-keys
Snowball Edge client commands. For more information, see Configuring and using the Snowball Edge Client. These local Amazon S3 credentials include a pair of keys: an access key and a secret key. These keys are only valid for the devices associated with your job. They can't be used in the Amazon Web Services Cloud because they have no Amazon Identity and Access Management (IAM) counterpart. -
The encryption key is not changed by what Amazon Web Services credentials you use. Signing with the Signature Version 4 algorithm is only used to verify the data traveling from its source to the interface. Thus, this signing never factors into the encryption keys used to encrypt your data on the Snowball.
Getting and using local Amazon S3 credentials on Snowball Edge
Every interaction with a Snowball Edge is signed with the Amazon Signature Version 4 algorithm. For more information about the algorithm, see Signature Version 4 Signing Process in the Amazon Web Services General Reference.
You can get the local Amazon S3 credentials to sign your requests to the Snowball Edge
client Edge device by running the snowballEdge list-access-keys
and
snowballEdge get-secret-access-key
Snowball Edge client information,
see Getting credentials for a Snowball Edge. These local
Amazon S3 credentials include a pair of keys: an access key ID and a secret key. These
credentials are only valid for the devices that are associated with your job. They can't
be used in the Amazon Web Services Cloud because they have no IAM counterpart.
You can add these credentials to the Amazon Web Services credentials file on your server. The default
credential profiles file is typically located at ~/.aws/credentials
, but
the location can vary per platform. This file is shared by many of the Amazon SDKs and by
the Amazon CLI. You can save local credentials with a profile name as in the following
example.
[snowballEdge] aws_access_key_id = AKIAIOSFODNN7EXAMPLE aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Configuring the Amazon CLI to use the S3 adapter on a Snowball Edge as the endpoint
When you use the Amazon CLI to issue a command to the Amazon Snowball Edge device, you specify that the endpoint is the Amazon S3 adapter. You have the choice of using the HTTPS endpoint, or an unsecured HTTP endpoint, as shown following.
HTTPS secured endpoint
aws s3 ls --profile snowballEdge --endpoint https://192.0.2.0:8443 --ca-bundle
path/to/certificate
HTTP unsecured endpoint
aws s3 ls --profile snowballEdge --endpoint http://192.0.2.0:8080
If you use the HTTPS endpoint of 8443
, your data is securely
transferred from your server to the Snowball Edge. This encryption is ensured with
a certificate that's generated by the Snowball Edge when it gets a new IP
address. After you have your certificate, you can save it to a local
ca-bundle.pem
file. Then you can configure your Amazon CLI profile to
include the path to your certificate, as described following.
To associate your certificate with the interface endpoint
-
Connect the Snowball Edge to power and the network, and turn it on.
-
After the device has finished booting up, make a note of its IP address on your local network.
-
From a terminal on your network, make sure you can ping the Snowball Edge.
-
Run the
snowballEdge get-certificate
command in your terminal. For more information on this command, see Managing public key certificates on Snowball Edge. -
Save the output of the
snowballEdge get-certificate
command to a file, for exampleca-bundle.pem
. -
Run the following command from your terminal.
aws configure set profile.snowballEdge.ca_bundle /path/to/ca-bundle.pem
After you complete the procedure, you can run CLI commands with these local credentials, your certificate, and your specified endpoint, as in the following example.
aws s3 ls --profile snowballEdge --endpoint https://192.0.2.0:8443
Unsupported Amazon S3 features for the Amazon S3 adapter on Snowball Edge
Using the Amazon S3 adapter, you can programmatically transfer data to and from a Snowball Edge with Amazon S3 API actions. However, not all Amazon S3 transfer features and API actions are supported for use with a Snowball Edge device when using the Amazon S3 adapter. For example, the following features and actions are not supported for use with Snowball Edge:
-
TransferManager – This utility transfers files from a local environment to Amazon S3 with the SDK for Java. Consider using the supported API actions or Amazon CLI commands with the interface instead.
-
GET Bucket (List Objects) Version 2 – This implementation of the GET action returns some or all (up to 1,000) of the objects in a bucket. Consider using the GET Bucket (List Objects) Version 1 action or the ls Amazon CLI command.
-
ListBuckets – The ListBuckets with the object endpoint is not supported. The following command does not work with Amazon S3 compatible storage on Snowball Edge:
aws s3 ls --endpoint
https://192.0.2.0
--profileprofile