Changes in Amazon S3 Transfer Manager from version 1 to version 2 - Amazon SDK for Java 2.x
High-level changesConfiguration API changesBehavior changes
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Changes in Amazon S3 Transfer Manager from version 1 to version 2

This topic details the changes in the Amazon S3 Transfer Manager from version 1 (v1) to version 2 (v2).

High-level changes

Change v1 v2

Maven dependencies

 
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bom</artifactId>
            <version>1.12.5871</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
    </dependency>
</dependencies>
 
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>bom</artifactId>
            <version>2.21.212</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>
<dependencies>
    <dependency>
        <groupId>software.amazon.awssdk</groupId>
        <artifactId>s3-transfer-manager</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.awssdk.crt</groupId>
        <artifactId>aws-crt</artifactId>
        <version>0.28.73</version>
    </dependency>
</dependencies>
Package name com.amazonaws.services.s3.transfer software.amazon.awssdk.transfer.s3
Class name

TransferManager

S3TransferManager

1 Latest version. 2 Latest version. 3Latest version.

Configuration API changes

Setting v1 v2

(get a builder)

 
TransferManagerBuilder tmBuilder = 
   TransferManagerBuilder.standard();
 
S3TransferManager.Builder tmBuilder  = 
  S3TransferManager.builder();

S3 client

 
tmBuilder.withS3Client(...);
tmBuilder.setS3Client(...);
 
tmBuilder.s3Client(...);

Executor

 
tmBuilder.withExecutorFactory(...);
tmBuilder.setExecutorFactory(...);
 
tmBuilder.executor(...);

Shutdown thread pools

 
tmBuilder.withShutDownThreadPools(...);
tmBuilder.setShutdownThreadPools(...);
 Not supported. The provided executor will not be shut down when the S3TransferManager is closed

Minimum upload part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 =
   S3AsyncClient.crtBuilder().
      minimumPartSizeInBytes(...).build();

tmBuilder.s3Client(s3);

Multipart upload threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 =
   S3AsyncClient.crtBuilder().
      thresholdInBytes(...).build();

tmBuilder.s3Client(s3);

Minimum copy part size

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 =
   S3AsyncClient.crtBuilder().
      minimumPartSizeInBytes(...).build();

tmBuilder.s3Client(s3);

Multipart copy threshold

 
tmBuilder.withMinimumUploadPartSize(...);
tmBuilder.setMinimumUploadPartSize(...);
 
S3AsyncClient s3 =
   S3AsyncClient.crtBuilder().
      thresholdInBytes(...).build();

tmBuilder.s3Client(s3);

Disable parallel downloads

 
tmBuilder.withDisableParallelDownloads(...);
tmBuilder.setDisableParallelDownloads(...);
 Disable parallel downloads by passing a standard Java-based S3 client to the transfer manager.
S3AsyncClient s3 =
   S3AsyncClient.builder().build();

tmBuilder.s3Client(s3);

Always calculate multipart md5

 
tmBuilder.withAlwaysCalculateMultipartMd5(...);
tmBuilder.setAlwaysCalculateMultipartMd5(...);
 Not supported.

Behavior changes

Parallel transfer requires Amazon CRT-based S3 client

In the SDK for Java 2.x, the automatic parallel transfer feature (multipart upload/download) is available through the Amazon CRT-based S3 client. To enable the parallel transfer feature , you must explicitly add the Amazon Common Runtime (CRT) library dependency for the maximized performance.

The Amazon CRT-based S3 client alone—without using S3TransferManager—provides maximized performance of parallel transfers. S3TransferManager v2 provides additional APIs that make it easier to transfer files and directories.

The ability for the S3TransferManager to perform parallel transfers depends on the how S3TransferManager is initiated and if the Amazon Common Runtime (CRT) library has been declared as a dependency.

The following table describes three initialization scenarios for an S3TransferManager v2 with and without the Amazon CRT declared as a dependency.

S3TransferManager v2 initialization approach Is Amazon CRT declared as a dependency?
yes no

Initialize the S3TransferManager without passing an S3AsyncClient instance

Static create method:

S3TransferManager.create();

- OR -

Builder method:

S3TransferManager.builder().build();
A green check mark indicating that automatic parallel transfer is enabled.

automatic parallel transfer enabled
A red X indicating that automatic parallel transfer is disabled.

automatic parallel transfer disabled

Pass an S3AsyncClient instance that is built with one of the crt*() builder methods

S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder().build();
S3TransferManager.builder().s3AsyncClient(s3AsyncClient).build();

- OR -

S3AsyncClient s3AsyncClient = S3AsyncClient.crtCreate();
S3TransferManager.builder().s3AsyncClient(s3AsyncClient).build();
A green check mark indicating that automatic parallel transfer is enabled.

automatic parallel transfer enabled
A red X indicating that automatic parallel transfer is disabled.

runtime error

Pass an S3AsyncClient instance that is built with one of the standard builder methods so that the transfer manager has no reference to the CRT

S3AsyncClient s3AsyncClient = S3AsyncClient.builder().build();
S3TransferManager.builder().s3AsyncClient(s3AsyncClient).build();

- OR -

S3AsyncClient s3AsyncClient = S3AsyncClient.create();
S3TransferManager.builder().s3AsyncClient(s3AsyncClient).build();
A red X indicating that automatic parallel transfer is disabled.

automatic parallel transfer disabled
A red X indicating that automatic parallel transfer is disabled.

automatic parallel transfer disabled

Parallel download via byte-range fetches

When the automatic parallel transfer feature is enabled, the S3 Transfer Manager v2 uses byte-range fetches to retrieve specific portions of the object in parallel (multipart download). The way an object is downloaded with v2 does not depend on how the object was originally uploaded. All downloads can benefit from high throughput and concurrency.

In contrast, with S3 Transfer Manager v1, it does matter how the object was originally uploaded. The S3 Transfer Manager v1 retrieves the parts of the object the same way that the parts were uploaded. If an object was originally uploaded as a single object, the S3 Transfer Manager v1 is not able to accelerate the downloading process by using sub-requests.

Failure behavior

With S3 Transfer Manager v1, a directory transfer request fails if any sub-request fails. Unlike v1, the future returned from S3 Transfer Manager v2 completes successfully even if some sub-requests fail.

As a result, you should check for errors in the response by using the CompletedDirectoryDownload.failedTransfers() method or CompletedDirectoryUpload.failedTransfers() method even when the future completes successfully.