Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (Amazon Signature Version 4)
As described in the Overview, when authenticating requests using the
Authorization
header, you have an option of uploading the payload in
chunks. You can send data in fixed size or variable size chunks. This section describes the
signature calculation process in chunked upload, how you create the chunk body, and how the
delayed signing works where you first upload the chunk, and send its signature in the
subsequent chunk. The example section (see Example: PUT Object) shows signature calculations
and resulting Authorization
headers that you can use as a test suite to verify
your code.
Note
When transferring data in a series of chunks, you must do one of the following:
-
Explicitly specify the total content length (object length in bytes plus metadata in each chunk) using the
Content-Length
HTTP header. To do this, you must pre-compute the total length of the payload, including the metadata that you send in each chunk, before starting your request. -
Specify the
Transfer-Encoding
HTTP header. If you include theTransfer-Encoding
header and specify any value other thanidentity
, you must omit theContent-Length
header.
For all requests, you must include the x-amz-decoded-content-length
header, specifying the size of the object in bytes.
Each chunk signature calculation includes the signature of the previous chunk. To begin, you create a seed signature using only the headers. You use the seed signature in the signature calculation of the first chunk. For each subsequent chunk, you create a chunk signature that includes the signature of the previous chunk. Thus, the chunk signatures are chained together; that is, the signature of chunk n is a function F(chunk n, signature(chunk n-1)). The chaining ensures that you send the chunks in the correct order.
To perform a chunked upload, do the following:
-
Decide the payload chunk size. You need this when you write the code.
The chunk size must be at least 8 KB. We recommend a chunk size of a least 64 KB for better performance. This chunk size applies to all chunks except the last one. The last chunk you send can be smaller than 8 KB. If your payload is small and can fit into one chunk, then it can be smaller than the 8 KB.
-
Create the seed signature for inclusion in the first chunk. For more information, see Calculating the Seed Signature.
-
Create the first chunk and stream it. For more information, see Defining the Chunk Body.
-
For each subsequent chunk, calculate the chunk signature that includes the previous signature in the string you sign, construct the chunk, and send it. For more information, see Defining the Chunk Body.
-
Send the final additional chunk, which is the same as the other chunks in the construction, but it has zero data bytes. For more information, see Defining the Chunk Body.
Calculating the Seed Signature
The following diagram illustrates the process of calculating the seed signature.
The following table describes the functions that are shown in the diagram. You need to implement code for these functions.
Function | Description |
---|---|
Lowercase() |
Convert the string to lowercase. |
Hex() |
Lowercase base 16 encoding. |
SHA256Hash() |
Secure Hash Algorithm (SHA) cryptographic hash function. |
HMAC-SHA256() |
Computes HMAC by using the SHA256 algorithm with the signing key provided. This is the final signature. |
Trim() |
Remove any leading or trailing whitespace. |
UriEncode() |
URI encode every byte. UriEncode() must enforce the following rules:
ImportantThe standard UriEncode functions provided by your development platform may not work because of differences in implementation and related ambiguity in the underlying RFCs. We recommend that you write your own custom UriEncode function to ensure that your encoding will work. To see an example of a UriEncode function in Java, see
Java Utilities |
For information about the signing process, see Signature Calculations for the Authorization Header:
Transferring Payload in a Single Chunk (Amazon Signature Version 4). The
process is the same, except that the creation of CanonicalRequest
differs
as follows:
-
In addition to the request headers you plan to add, you must include the following headers:
Header Description x-amz-content-sha256
This header is required for all Amazon Signature Version 4 requests. Set the value to
STREAMING-AWS4-HMAC-SHA256-PAYLOAD
to indicate that the signature covers only headers and that there is no payload.Content-Encoding
Set the value to
aws-chunked
.Amazon S3 supports multiple content encodings. For example:
Content-Encoding : aws-chunked,gzip
That is, you can specify your custom content-encoding when using Signature Version 4 streaming API.
If you specify
Content-Encoding
in your request asContent-Encoding : aws-chunked
, S3 adds an empty value forContent-Encoding
and stores the object metadata (Content-Encoding :
) to the resulting object.Note
Amazon S3 stores the resulting object without the
aws-chunked
encoding. Therefore, when you retrieve the object, it is notaws-chunked
encoded.x-amz-decoded-content-length
Set the value to the length, in bytes, of the data to be chunked, without counting any metadata. For example, if you are uploading a 4 GB file, set the value to 4294967296. This is the raw size of the object to be uploaded (data you want to store in Amazon S3). Content-Length
Set the value to the actual size of the transmitted HTTP body, which includes the length of your data (value set for
x-amz-decoded-content-length
), plus chunk metadata. Each chunk has metadata, such as the signature of the previous chunk. Chunk calculations are discussed in the following section. If you include theTransfer-Encoding
header and specify any value other thanidentity
, you must not include theContent-Length
header.
You send the first chunk with the seed signature. You must construct the chunk as described in the following section.
Defining the Chunk Body
All chunks include some metadata. Each chunk must conform to the following structure:
string(IntHexBase(
chunk-size
)) + ";chunk-signature=" +signature
+ \r\n +chunk-data
+ \r\n
Where:
-
IntHexBase()
is a function that you write to convert an integer chunk-size to hexadecimal. For example, if chunk-size is 65536, hexadecimal string is "10000". -
chunk-size
is the size, in bytes, of the chunk-data, without metadata. For example, if you are uploading a 65 KB object and using a chunk size of 64 KB, you upload the data in three chunks: the first would be 64 KB, the second 1 KB, and the final chunk with 0 bytes. -
signature
For each chunk, you calculate the signature using the following string to sign. For the first chunk, you use the seed-signature as the previous signature.
The size of the final chunk data that you send is 0, although the chunk body still contains metadata, including the signature of the previous chunk.
Example: PUT Object
You can use the examples in this section as a reference to check signature calculations in your code. Before you review the examples, note the following:
-
The signature calculations in these examples use the following example security credentials.
Parameter Value AWSAccessKeyId
AKIAIOSFODNN7EXAMPLE
AWSSecretAccessKey
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
-
All examples use the request timestamp 20130524T000000Z (
Fri, 24 May 2013 00:00:00 GMT
). -
All examples use
examplebucket
as the bucket name. -
The bucket is assumed to be in the US East (N. Virginia) Region, and the credential
Scope
and theSigning Key
calculations useus-east-1
as the Region specifier. For more information, see Regions and Endpoints in the Amazon Web Services General Reference. -
You can use either path style or virtual-hosted style requests. The following examples use virtual-hosted style requests, for example:
https://examplebucket.s3.amazonaws.com/photos/photo1.jpg
For more information, see Virtual Hosting of Buckets in the Amazon Simple Storage Service User Guide.
The following example sends a PUT
request to upload an object. The
signature calculations assume the following:
-
You are uploading a 65 KB text file, and the file content is a one-character string made up of the letter 'a'.
-
The chunk size is 64 KB. As a result, the payload is uploaded in three chunks, 64 KB, 1 KB, and the final chunk with 0 bytes of chunk data.
-
The resulting object has the key name
chunkObject.txt
. -
You are requesting
REDUCED_REDUNDANCY
as the storage class by adding thex-amz-storage-class
request header.
For information about the API action, see PutObject. The general request syntax is as follows:
PUT /examplebucket/chunkObject.txt HTTP/1.1 Host: s3.amazonaws.com x-amz-date: 20130524T000000Z x-amz-storage-class: REDUCED_REDUNDANCY Authorization:
SignatureToBeCalculated
x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD Content-Encoding: aws-chunked x-amz-decoded-content-length: 66560 Content-Length: 66824<Payload>
The following steps show signature calculations.
-
Seed signature — Create String to Sign
-
CanonicalRequest
PUT /examplebucket/chunkObject.txt content-encoding:aws-chunked content-length:66824 host:s3.amazonaws.com x-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOAD x-amz-date:20130524T000000Z x-amz-decoded-content-length:66560 x-amz-storage-class:REDUCED_REDUNDANCY content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class STREAMING-AWS4-HMAC-SHA256-PAYLOAD
In the canonical request, the third line is empty because there are no query parameters in the request. The last line is the constant string provided as the value of the hashed Payload, which should be same as the value of
x-amz-content-sha256 header
. -
StringToSign
AWS4-HMAC-SHA256 20130524T000000Z 20130524/us-east-1/s3/aws4_request cee3fed04b70f867d036f722359b0b1f2f0e5dc0efadbc082b76c4c60e316455
Note
For information about each of line in the string to sign, see the diagram that explains seed signature calculation.
-
-
SigningKey
signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "
<YourSecretAccessKey>
","20130524"),"us-east-1"),"s3"),"aws4_request") -
Seed Signature
4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9
-
Authorization header
The resulting Authorization header is as follows:
AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9
-
Chunk 1: (65536 bytes, with value 97 for letter 'a')
-
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request 4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 bf718b6f653bebc184e1479f1935b8da974d701b893afcf49e701f3e2f9f9c5a
Note
For information about each line in the string to sign, see the preceding diagram that shows various components of the string to sign (for example, the last three lines are,
previous-signature
,hash("")
, andhash(current-chunk-data)
). -
Chunk signature:
ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648
-
Chunk data sent:
10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648 <65536-bytes>
-
-
Chunk 2: (1024 bytes, with value 97 for letter 'a')
-
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4a
-
Chunk signature:
0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497
-
Chunk data sent:
400;chunk-signature=0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497 <1024 bytes>
-
-
Chunk 3: (0 byte data)
-
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD 20130524T000000Z 20130524/us-east-1/s3/aws4_request 0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
-
Chunk signature:
b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9
-
Chunk data sent:
0;chunk-signature=b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9
-