使用 Amazon SDK for .NET 在 Amazon S3 Glacier 中下载档案 - Amazon S3 Glacier
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

如果您不熟悉 Amazon Simple Storage Service (Amazon S3) 中的归档存储功能,建议您先详细了解 Amazon S3 中的 S3 Glacier 存储类、S3 Glacier 即时检索S3 Glacier 灵活检索S3 Glacier 深度归档。有关更多信息,请参阅 Amazon S3 用户指南中的 S3 Glacier 存储类和用于存档对象的存储类。

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

使用 Amazon SDK for .NET 在 Amazon S3 Glacier 中下载档案

适用于 .NET 的 Amazon SDK 提供的高级和低级 API 都提供了下载档案的方法。

使用Amazon SDK for .NET高级 API 下载档案

该高级 API 的 ArchiveTransferManager 类提供了您可以用来下载档案的 Download 方法。

重要

ArchiveTransferManager 类将创建 Amazon Simple Notification Service (Amazon SNS) 主题,以及该主题订阅的 Amazon Simple Queue Service (Amazon SQS) 队列。然后,它启动了档案检索任务,并对队列进行轮询以便找到可用档案。如果存档可用,则开始下载。有关检索时间的详细信息,请参阅 档案检索选项

示例:使用Amazon SDK for .NET的高级 API 下载档案

以下 C# 代码示例将从美国西部(俄勒冈州)区域中的文件库 (examplevault) 中下载档案。

有关如何运行以下示例的分步说明,请参阅运行代码示例。您需要更新现有档案 ID 和已下载档案的本地文件保存路径旁显示的代码

using System; using Amazon.Glacier; using Amazon.Glacier.Transfer; using Amazon.Runtime; namespace glacier.amazon.com.docsamples { class ArchiveDownloadHighLevel { static string vaultName = "examplevault"; static string archiveId = "*** Provide archive ID ***"; static string downloadFilePath = "*** Provide the file name and path to where to store the download ***"; public static void Main(string[] args) { try { var manager = new ArchiveTransferManager(Amazon.RegionEndpoint.USWest2); var options = new DownloadOptions(); options.StreamTransferProgress += ArchiveDownloadHighLevel.progress; // Download an archive. Console.WriteLine("Intiating the archive retrieval job and then polling SQS queue for the archive to be available."); Console.WriteLine("Once the archive is available, downloading will begin."); manager.Download(vaultName, archiveId, downloadFilePath, options); Console.WriteLine("To continue, press Enter"); Console.ReadKey(); } catch (AmazonGlacierException e) { Console.WriteLine(e.Message); } catch (AmazonServiceException e) { Console.WriteLine(e.Message); } catch (Exception e) { Console.WriteLine(e.Message); } Console.WriteLine("To continue, press Enter"); Console.ReadKey(); } static int currentPercentage = -1; static void progress(object sender, StreamTransferProgressArgs args) { if (args.PercentDone != currentPercentage) { currentPercentage = args.PercentDone; Console.WriteLine("Downloaded {0}%", args.PercentDone); } } } }

使用Amazon SDK for .NET低级 API 下载档案

以下是使用Amazon SDK for .NET的低级 API 下载 Amazon S3 Glacier (S3 Glacier) 档案的步骤。

  1. 创建 AmazonGlacierClient 类(客户端)的实例。

    您需要指定要从中下载档案的 Amazon 区域。您使用此客户端执行的所有操作都会应用到该 Amazon 区域。

  2. 通过执行 archive-retrieval 方法启动 InitiateJob 任务。

    您可以通过创建一个 InitiateJobRequest 类的实例提供任务信息,例如,您要下载的档案的档案 ID,以及您希望 S3 Glacier 向其发布任务完成消息的可选 Amazon SNS 主题。作为响应,S3 Glacier 返回任务 ID。该响应位于一个 InitiateJobResponse 类的实例中。

    AmazonGlacierClient client; client = new AmazonGlacierClient(Amazon.RegionEndpoint.USWest2); InitiateJobRequest initJobRequest = new InitiateJobRequest() { VaultName = vaultName, JobParameters = new JobParameters() { Type = "archive-retrieval", ArchiveId = "*** Provide archive id ***", SNSTopic = "*** Provide Amazon SNS topic ARN ***", } }; InitiateJobResponse initJobResponse = client.InitiateJob(initJobRequest); string jobId = initJobResponse.JobId;

    您可以选择指定字节范围,以请求 S3 Glacier 只准备档案的一部分,如以下请求所示。该请求指定 S3 Glacier 只准备档案的 1 MB 到 2 MB 部分。

    AmazonGlacierClient client; client = new AmazonGlacierClient(Amazon.RegionEndpoint.USWest2); InitiateJobRequest initJobRequest = new InitiateJobRequest() { VaultName = vaultName, JobParameters = new JobParameters() { Type = "archive-retrieval", ArchiveId = "*** Provide archive id ***", SNSTopic = "*** Provide Amazon SNS topic ARN ***", } }; // Specify byte range. int ONE_MEG = 1048576; initJobRequest.JobParameters.RetrievalByteRange = string.Format("{0}-{1}", ONE_MEG, 2 * ONE_MEG -1); InitiateJobResponse initJobResponse = client.InitiateJob(initJobRequest); string jobId = initJobResponse.JobId;
  3. 等待任务完成。

    您必须等到任务输出已作好供您下载的准备。如果您在文件库中设置了标识 Amazon Simple Notification Service (Amazon SNS) 主题的通知配置,或者在启动任务时指定了Amazon SNS 主题,则 S3 Glacier 会在完成任务后向该主题发送消息。以下部分给出的代码示例使用适用于 S3 Glacier 的 Amazon SNS 来发布消息。

    此外,您还可以通过调用 DescribeJob 方法轮询 S3 Glacier 来确定任务完成状态。尽管如此,使用 Amazon SNS 主题进行通知才是推荐的方法。

  4. 通过执行 GetJobOutput 方法下载任务输出(档案数据)。

    您可以通过创建一个 GetJobOutputRequest 类的实例来提供请求信息,例如,任务 ID 和文件库名称。S3 Glacier 返回的输出位于 GetJobOutputResponse 对象中。

    GetJobOutputRequest getJobOutputRequest = new GetJobOutputRequest() { JobId = jobId, VaultName = vaultName }; GetJobOutputResponse getJobOutputResponse = client.GetJobOutput(getJobOutputRequest); using (Stream webStream = getJobOutputResponse.Body) { using (Stream fileToSave = File.OpenWrite(fileName)) { CopyStream(webStream, fileToSave); } }

    前面的代码段会下载整个任务输出。您可以选择性地只检索输出的一部分,或者通过在您的 GetJobOutputRequest 中指定字节范围以较小的区块下载整个输出。

    GetJobOutputRequest getJobOutputRequest = new GetJobOutputRequest() { JobId = jobId, VaultName = vaultName }; getJobOutputRequest.SetRange(0, 1048575); // Download only the first 1 MB chunk of the output.

    在响应您的 GetJobOutput 调用时,S3 Glacier 返回您下载的数据部分的校验和(如果满足特定条件)。有关更多信息,请参阅下载数据时接收校验和

    要确认下载中没有错误,您随后可以在客户端计算校验和,并将它与 S3 Glacier 在响应中发送的校验和相比较。

    对于指定了可选范围的档案检索任务,您在获取任务描述时,它会包括您要检索的范围的校验和 (SHA256TreeHash)。您可以使用此值进一步确认您稍后下载的整个字节范围的准确性。例如,如果您启动任务以检索以树形哈希对齐的档案范围,然后以区块下载输出,使得您的每个 GetJobOutput 请求均返回一个校验和,则您可以在客户端计算您下载的每个部分的校验和,然后计算树形哈希。您可以将它与 S3 Glacier 为响应您的描述任务请求而返回的校验和相比较,以确认您下载的整个字节范围与 S3 Glacier 中存储的字节范围相同。

    要了解可工作的示例,请参阅 示例 2:使用 Amazon SDK for .NET低级 API 检索档案— 按区块下载输出

示例 1:使用Amazon SDK for .NET低级 API 检索档案

以下 C# 代码示例会从指定的文件库下载档案。任务完成后,该示例会在单一 GetJobOutput 调用中下载整个输出。有关以区块下载输出的示例,请参阅“示例 2:使用 Amazon SDK for .NET低级 API 检索档案— 按区块下载输出”。

该示例执行以下任务:

  • 设置 Amazon Simple Notification Service (Amazon SNS) 主题

    完成任务后,S3 Glacier 会向此主题发送通知。

  • 设置 Amazon Simple Queue Service (Amazon SQS) 队列

    该示例会向该队列附加策略,以使 Amazon SNS 主题能够发布消息。

  • 启动任务以下载指定的档案。

    在任务请求中,该示例会指定 Amazon SNS 主题,以便 S3 Glacier 可以在完成任务后发送消息。

  • 定期检查 Amazon SQS 队列是否有消息。

    如果有消息,则分析 JSON,并检查任务是否已成功完成。如果已成功完成,则下载档案。该代码示例使用 JSON.NET 库(请参阅 JSON.NET)来分析 JSON。

  • 通过删除它创建的 Amazon SNS 主题和 Amazon SQS 队列清除相关数据。

using System; using System.Collections.Generic; using System.IO; using System.Threading; using Amazon.Glacier; using Amazon.Glacier.Model; using Amazon.Runtime; using Amazon.SimpleNotificationService; using Amazon.SimpleNotificationService.Model; using Amazon.SQS; using Amazon.SQS.Model; using Newtonsoft.Json; namespace glacier.amazon.com.docsamples { class ArchiveDownloadLowLevelUsingSNSSQS { static string topicArn; static string queueUrl; static string queueArn; static string vaultName = "*** Provide vault name ***"; static string archiveID = "*** Provide archive ID ***"; static string fileName = "*** Provide the file name and path to where to store downloaded archive ***"; static AmazonSimpleNotificationServiceClient snsClient; static AmazonSQSClient sqsClient; const string SQS_POLICY = "{" + " \"Version\" : \"2012-10-17\"," + " \"Statement\" : [" + " {" + " \"Sid\" : \"sns-rule\"," + " \"Effect\" : \"Allow\"," + " \"Principal\" : {\"Service\" : \"sns.amazonaws.com\" }," + " \"Action\" : \"sqs:SendMessage\"," + " \"Resource\" : \"{QueueArn}\"," + " \"Condition\" : {" + " \"ArnLike\" : {" + " \"aws:SourceArn\" : \"{TopicArn}\"" + " }" + " }" + " }" + " ]" + "}"; public static void Main(string[] args) { AmazonGlacierClient client; try { using (client = new AmazonGlacierClient(Amazon.RegionEndpoint.USWest2)) { Console.WriteLine("Setup SNS topic and SQS queue."); SetupTopicAndQueue(); Console.WriteLine("To continue, press Enter"); Console.ReadKey(); Console.WriteLine("Retrieving..."); RetrieveArchive(client); } Console.WriteLine("Operations successful. To continue, press Enter"); Console.ReadKey(); } catch (AmazonGlacierException e) { Console.WriteLine(e.Message); } catch (AmazonServiceException e) { Console.WriteLine(e.Message); } catch (Exception e) { Console.WriteLine(e.Message); } finally { // Delete SNS topic and SQS queue. snsClient.DeleteTopic(new DeleteTopicRequest() { TopicArn = topicArn }); sqsClient.DeleteQueue(new DeleteQueueRequest() { QueueUrl = queueUrl }); } } static void SetupTopicAndQueue() { snsClient = new AmazonSimpleNotificationServiceClient(Amazon.RegionEndpoint.USWest2); sqsClient = new AmazonSQSClient(Amazon.RegionEndpoint.USWest2); long ticks = DateTime.Now.Ticks; topicArn = snsClient.CreateTopic(new CreateTopicRequest { Name = "GlacierDownload-" + ticks }).TopicArn; Console.Write("topicArn: "); Console.WriteLine(topicArn); CreateQueueRequest createQueueRequest = new CreateQueueRequest(); createQueueRequest.QueueName = "GlacierDownload-" + ticks; CreateQueueResponse createQueueResponse = sqsClient.CreateQueue(createQueueRequest); queueUrl = createQueueResponse.QueueUrl; Console.Write("QueueURL: "); Console.WriteLine(queueUrl); GetQueueAttributesRequest getQueueAttributesRequest = new GetQueueAttributesRequest(); getQueueAttributesRequest.AttributeNames = new List<string> { "QueueArn" }; getQueueAttributesRequest.QueueUrl = queueUrl; GetQueueAttributesResponse response = sqsClient.GetQueueAttributes(getQueueAttributesRequest); queueArn = response.QueueARN; Console.Write("QueueArn: "); Console.WriteLine(queueArn); // Setup the Amazon SNS topic to publish to the SQS queue. snsClient.Subscribe(new SubscribeRequest() { Protocol = "sqs", Endpoint = queueArn, TopicArn = topicArn }); // Add policy to the queue so SNS can send messages to the queue. var policy = SQS_POLICY.Replace("{TopicArn}", topicArn).Replace("{QueueArn}", queueArn); sqsClient.SetQueueAttributes(new SetQueueAttributesRequest() { QueueUrl = queueUrl, Attributes = new Dictionary<string, string> { { QueueAttributeName.Policy, policy } } }); } static void RetrieveArchive(AmazonGlacierClient client) { // Initiate job. InitiateJobRequest initJobRequest = new InitiateJobRequest() { VaultName = vaultName, JobParameters = new JobParameters() { Type = "archive-retrieval", ArchiveId = archiveID, Description = "This job is to download archive.", SNSTopic = topicArn, } }; InitiateJobResponse initJobResponse = client.InitiateJob(initJobRequest); string jobId = initJobResponse.JobId; // Check queue for a message and if job completed successfully, download archive. ProcessQueue(jobId, client); } private static void ProcessQueue(string jobId, AmazonGlacierClient client) { ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest() { QueueUrl = queueUrl, MaxNumberOfMessages = 1 }; bool jobDone = false; while (!jobDone) { Console.WriteLine("Poll SQS queue"); ReceiveMessageResponse receiveMessageResponse = sqsClient.ReceiveMessage(receiveMessageRequest); if (receiveMessageResponse.Messages.Count == 0) { Thread.Sleep(10000 * 60); continue; } Console.WriteLine("Got message"); Message message = receiveMessageResponse.Messages[0]; Dictionary<string, string> outerLayer = JsonConvert.DeserializeObject<Dictionary<string, string>>(message.Body); Dictionary<string, object> fields = JsonConvert.DeserializeObject<Dictionary<string, object>>(outerLayer["Message"]); string statusCode = fields["StatusCode"] as string; if (string.Equals(statusCode, GlacierUtils.JOB_STATUS_SUCCEEDED, StringComparison.InvariantCultureIgnoreCase)) { Console.WriteLine("Downloading job output"); DownloadOutput(jobId, client); // Save job output to the specified file location. } else if (string.Equals(statusCode, GlacierUtils.JOB_STATUS_FAILED, StringComparison.InvariantCultureIgnoreCase)) Console.WriteLine("Job failed... cannot download the archive."); jobDone = true; sqsClient.DeleteMessage(new DeleteMessageRequest() { QueueUrl = queueUrl, ReceiptHandle = message.ReceiptHandle }); } } private static void DownloadOutput(string jobId, AmazonGlacierClient client) { GetJobOutputRequest getJobOutputRequest = new GetJobOutputRequest() { JobId = jobId, VaultName = vaultName }; GetJobOutputResponse getJobOutputResponse = client.GetJobOutput(getJobOutputRequest); using (Stream webStream = getJobOutputResponse.Body) { using (Stream fileToSave = File.OpenWrite(fileName)) { CopyStream(webStream, fileToSave); } } } public static void CopyStream(Stream input, Stream output) { byte[] buffer = new byte[65536]; int length; while ((length = input.Read(buffer, 0, buffer.Length)) > 0) { output.Write(buffer, 0, length); } } } }

示例 2:使用 Amazon SDK for .NET低级 API 检索档案— 按区块下载输出

以下 C# 代码示例从 S3 Glacier 检索档案。该代码示例会通过在 GetJobOutputRequest 数据元中指定字节范围来以区块下载任务输出。

using System; using System.Collections.Generic; using System.IO; using System.Threading; using Amazon.Glacier; using Amazon.Glacier.Model; using Amazon.Glacier.Transfer; using Amazon.Runtime; using Amazon.SimpleNotificationService; using Amazon.SimpleNotificationService.Model; using Amazon.SQS; using Amazon.SQS.Model; using Newtonsoft.Json; using System.Collections.Specialized; namespace glacier.amazon.com.docsamples { class ArchiveDownloadLowLevelUsingSQLSNSOutputUsingRange { static string topicArn; static string queueUrl; static string queueArn; static string vaultName = "*** Provide vault name ***"; static string archiveId = "*** Provide archive ID ***"; static string fileName = "*** Provide the file name and path to where to store downloaded archive ***"; static AmazonSimpleNotificationServiceClient snsClient; static AmazonSQSClient sqsClient; const string SQS_POLICY = "{" + " \"Version\" : \"2012-10-17\"," + " \"Statement\" : [" + " {" + " \"Sid\" : \"sns-rule\"," + " \"Effect\" : \"Allow\"," + " \"Principal\" : {\"AWS\" : \"arn:aws:iam::123456789012:root\" }," + " \"Action\" : \"sqs:SendMessage\"," + " \"Resource\" : \"{QuernArn}\"," + " \"Condition\" : {" + " \"ArnLike\" : {" + " \"aws:SourceArn\" : \"{TopicArn}\"" + " }" + " }" + " }" + " ]" + "}"; public static void Main(string[] args) { AmazonGlacierClient client; try { using (client = new AmazonGlacierClient(Amazon.RegionEndpoint.USWest2)) { Console.WriteLine("Setup SNS topic and SQS queue."); SetupTopicAndQueue(); Console.WriteLine("To continue, press Enter"); Console.ReadKey(); Console.WriteLine("Download archive"); DownloadAnArchive(archiveId, client); } Console.WriteLine("Operations successful. To continue, press Enter"); Console.ReadKey(); } catch (AmazonGlacierException e) { Console.WriteLine(e.Message); } catch (AmazonServiceException e) { Console.WriteLine(e.Message); } catch (Exception e) { Console.WriteLine(e.Message); } finally { // Delete SNS topic and SQS queue. snsClient.DeleteTopic(new DeleteTopicRequest() { TopicArn = topicArn }); sqsClient.DeleteQueue(new DeleteQueueRequest() { QueueUrl = queueUrl }); } } static void SetupTopicAndQueue() { long ticks = DateTime.Now.Ticks; // Setup SNS topic. snsClient = new AmazonSimpleNotificationServiceClient(Amazon.RegionEndpoint.USWest2); sqsClient = new AmazonSQSClient(Amazon.RegionEndpoint.USWest2); topicArn = snsClient.CreateTopic(new CreateTopicRequest { Name = "GlacierDownload-" + ticks }).TopicArn; Console.Write("topicArn: "); Console.WriteLine(topicArn); CreateQueueRequest createQueueRequest = new CreateQueueRequest(); createQueueRequest.QueueName = "GlacierDownload-" + ticks; CreateQueueResponse createQueueResponse = sqsClient.CreateQueue(createQueueRequest); queueUrl = createQueueResponse.QueueUrl; Console.Write("QueueURL: "); Console.WriteLine(queueUrl); GetQueueAttributesRequest getQueueAttributesRequest = new GetQueueAttributesRequest(); getQueueAttributesRequest.AttributeNames = new List<string> { "QueueArn" }; getQueueAttributesRequest.QueueUrl = queueUrl; GetQueueAttributesResponse response = sqsClient.GetQueueAttributes(getQueueAttributesRequest); queueArn = response.QueueARN; Console.Write("QueueArn: "); Console.WriteLine(queueArn); // Setup the Amazon SNS topic to publish to the SQS queue. snsClient.Subscribe(new SubscribeRequest() { Protocol = "sqs", Endpoint = queueArn, TopicArn = topicArn }); // Add the policy to the queue so SNS can send messages to the queue. var policy = SQS_POLICY.Replace("{TopicArn}", topicArn).Replace("{QuernArn}", queueArn); sqsClient.SetQueueAttributes(new SetQueueAttributesRequest() { QueueUrl = queueUrl, Attributes = new Dictionary<string, string> { { QueueAttributeName.Policy, policy } } }); } static void DownloadAnArchive(string archiveId, AmazonGlacierClient client) { // Initiate job. InitiateJobRequest initJobRequest = new InitiateJobRequest() { VaultName = vaultName, JobParameters = new JobParameters() { Type = "archive-retrieval", ArchiveId = archiveId, Description = "This job is to download the archive.", SNSTopic = topicArn, } }; InitiateJobResponse initJobResponse = client.InitiateJob(initJobRequest); string jobId = initJobResponse.JobId; // Check queue for a message and if job completed successfully, download archive. ProcessQueue(jobId, client); } private static void ProcessQueue(string jobId, AmazonGlacierClient client) { var receiveMessageRequest = new ReceiveMessageRequest() { QueueUrl = queueUrl, MaxNumberOfMessages = 1 }; bool jobDone = false; while (!jobDone) { Console.WriteLine("Poll SQS queue"); ReceiveMessageResponse receiveMessageResponse = sqsClient.ReceiveMessage(receiveMessageRequest); if (receiveMessageResponse.Messages.Count == 0) { Thread.Sleep(10000 * 60); continue; } Console.WriteLine("Got message"); Message message = receiveMessageResponse.Messages[0]; Dictionary<string, string> outerLayer = JsonConvert.DeserializeObject<Dictionary<string, string>>(message.Body); Dictionary<string, object> fields = JsonConvert.DeserializeObject<Dictionary<string, object>>(outerLayer["Message"]); string statusCode = fields["StatusCode"] as string; if (string.Equals(statusCode, GlacierUtils.JOB_STATUS_SUCCEEDED, StringComparison.InvariantCultureIgnoreCase)) { long archiveSize = Convert.ToInt64(fields["ArchiveSizeInBytes"]); Console.WriteLine("Downloading job output"); DownloadOutput(jobId, archiveSize, client); // This where we save job output to the specified file location. } else if (string.Equals(statusCode, GlacierUtils.JOB_STATUS_FAILED, StringComparison.InvariantCultureIgnoreCase)) Console.WriteLine("Job failed... cannot download the archive."); jobDone = true; sqsClient.DeleteMessage(new DeleteMessageRequest() { QueueUrl = queueUrl, ReceiptHandle = message.ReceiptHandle }); } } private static void DownloadOutput(string jobId, long archiveSize, AmazonGlacierClient client) { long partSize = 4 * (long)Math.Pow(2, 20); // 4 MB. using (Stream fileToSave = new FileStream(fileName, FileMode.Create, FileAccess.Write)) { long currentPosition = 0; do { GetJobOutputRequest getJobOutputRequest = new GetJobOutputRequest() { JobId = jobId, VaultName = vaultName }; long endPosition = currentPosition + partSize - 1; if (endPosition > archiveSize) endPosition = archiveSize; getJobOutputRequest.SetRange(currentPosition, endPosition); GetJobOutputResponse getJobOutputResponse = client.GetJobOutput(getJobOutputRequest); using (Stream webStream = getJobOutputResponse.Body) { CopyStream(webStream, fileToSave); } currentPosition += partSize; } while (currentPosition < archiveSize); } } public static void CopyStream(Stream input, Stream output) { byte[] buffer = new byte[65536]; int length; while ((length = input.Read(buffer, 0, buffer.Length)) > 0) { output.Write(buffer, 0, length); } } } }