使用 Java 和 Amazon S3 管理大型亚马逊 SQS 消息 - Amazon Simple Queue Service
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

使用 Java 和 Amazon S3 管理大型亚马逊 SQS 消息

然后,您可以使用适用于 Java 的 Amazon SQS 扩展客户端库和亚马逊简单存储服务 (Amazon S3) Service 来管理大型亚马逊简单队列服务 (Amazon SQS) Simple Queue Service 消息。这对于消耗从 256 KB 到 2 GB 的大型消息负载特别有用。该库将消息负载保存到 Amazon S3 存储桶,并将包含存储的 Amazon S3 对象引用的消息发送到亚马逊 SQS 队列。

您可以使用适用于 Java 的 Amazon SQS 扩展型客户端库执行以下操作:

  • 指定消息是始终存储在 Amazon S3 中还是仅在其大小超过 256 KB 时存储在 Amazon S3 中。

  • 发送引用存储在 S3 存储桶中的单个消息对象的消息

  • 从 S3 存储桶检索消息对象

  • 从 S3 存储桶中删除消息对象

先决条件

以下示例使用的是 Amazon Java 开发工具包。要安装和设置开发工具包,请参阅 Amazon SDK for Java 开发人员指南中的设置 Amazon SDK for Java

运行示例代码之前,请配置您的 Amazon 凭证。有关更多信息,请参阅《Amazon SDK for Java 开发人员指南》中的设置 Amazon 凭据和开发区域

适用于 Java 的 SDK 和适用于 Java 的 Amazon SQS 扩展型客户端库需要使用 J2SE Development Kit 8.0 或更高版本。

注意

您可以通过适用于 Java 的 Amazon SQS 扩展型客户端库,利用 Amazon S3 与Amazon SDK for Java 来管理 Amazon SQS 消息。您无法使用 Amazon CLI、Amazon SQS 控制台、Amazon SQS HTTP API 或任何其他 Amazon SDK,执行此操作。

Amazon适用于 Java 的 SDK 1.x 示例:使用亚马逊 S3 管理大型亚马逊 SQS 消息

以下Amazon适用于 Java 的 SDK 2.x 示例创建了一个名称随机的 Amazon S3 存储桶,并添加了一条生命周期规则,用于在 14 天后永久删除对象。它还会创建一个名为 MyQueue 的队列并向该队列发送一条存储在 S3 存储桶中且大小超过 256 KB 的随机消息。最后,代码会检索该消息,返回该消息的相关信息,并删除该消息、队列和存储桶。

/* * Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"). * You may not use this file except in compliance with the License. * A copy of the License is located at * * https://aws.amazon.com/apache2.0 * * or in the "license" file accompanying this file. This file is distributed * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either * express or implied. See the License for the specific language governing * permissions and limitations under the License. * */ import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient; import com.amazon.sqs.javamessaging.ExtendedClientConfiguration; import com.amazonaws.services.s3.AmazonS3; import com.amazonaws.services.s3.AmazonS3ClientBuilder; import com.amazonaws.services.s3.model.*; import com.amazonaws.services.sqs.AmazonSQS; import com.amazonaws.services.sqs.AmazonSQSClientBuilder; import com.amazonaws.services.sqs.model.*; import org.joda.time.DateTime; import org.joda.time.format.DateTimeFormat; import java.util.Arrays; import java.util.List; import java.util.UUID; public class SQSExtendedClientExample { // Create an Amazon S3 bucket with a random name. private final static String S3_BUCKET_NAME = UUID.randomUUID() + "-" + DateTimeFormat.forPattern("yyMMdd-hhmmss").print(new DateTime()); public static void main(String[] args) { /* * Create a new instance of the builder with all defaults (credentials * and region) set automatically. For more information, see * Creating Service Clients in the Amazon SDK for Java Developer Guide. */ final AmazonS3 s3 = AmazonS3ClientBuilder.defaultClient(); /* * Set the Amazon S3 bucket name, and then set a lifecycle rule on the * bucket to permanently delete objects 14 days after each object's * creation date. */ final BucketLifecycleConfiguration.Rule expirationRule = new BucketLifecycleConfiguration.Rule(); expirationRule.withExpirationInDays(14).withStatus("Enabled"); final BucketLifecycleConfiguration lifecycleConfig = new BucketLifecycleConfiguration().withRules(expirationRule); // Create the bucket and allow message objects to be stored in the bucket. s3.createBucket(S3_BUCKET_NAME); s3.setBucketLifecycleConfiguration(S3_BUCKET_NAME, lifecycleConfig); System.out.println("Bucket created and configured."); /* * Set the Amazon SQS extended client configuration with large payload * support enabled. */ final ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration() .withLargePayloadSupportEnabled(s3, S3_BUCKET_NAME); final AmazonSQS sqsExtended = new AmazonSQSExtendedClient(AmazonSQSClientBuilder .defaultClient(), extendedClientConfig); /* * Create a long string of characters for the message object which will * be stored in the bucket. */ int stringLength = 300000; char[] chars = new char[stringLength]; Arrays.fill(chars, 'x'); final String myLongString = new String(chars); // Create a message queue for this example. final String QueueName = "MyQueue" + UUID.randomUUID().toString(); final CreateQueueRequest createQueueRequest = new CreateQueueRequest(QueueName); final String myQueueUrl = sqsExtended .createQueue(createQueueRequest).getQueueUrl(); System.out.println("Queue created."); // Send the message. final SendMessageRequest myMessageRequest = new SendMessageRequest(myQueueUrl, myLongString); sqsExtended.sendMessage(myMessageRequest); System.out.println("Sent the message."); // Receive the message. final ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(myQueueUrl); List<Message> messages = sqsExtended .receiveMessage(receiveMessageRequest).getMessages(); // Print information about the message. for (Message message : messages) { System.out.println("\nMessage received."); System.out.println(" ID: " + message.getMessageId()); System.out.println(" Receipt handle: " + message.getReceiptHandle()); System.out.println(" Message body (first 5 characters): " + message.getBody().substring(0, 5)); } // Delete the message, the queue, and the bucket. final String messageReceiptHandle = messages.get(0).getReceiptHandle(); sqsExtended.deleteMessage(new DeleteMessageRequest(myQueueUrl, messageReceiptHandle)); System.out.println("Deleted the message."); sqsExtended.deleteQueue(new DeleteQueueRequest(myQueueUrl)); System.out.println("Deleted the queue."); deleteBucketAndAllContents(s3); System.out.println("Deleted the bucket."); } private static void deleteBucketAndAllContents(AmazonS3 client) { ObjectListing objectListing = client.listObjects(S3_BUCKET_NAME); while (true) { for (S3ObjectSummary objectSummary : objectListing .getObjectSummaries()) { client.deleteObject(S3_BUCKET_NAME, objectSummary.getKey()); } if (objectListing.isTruncated()) { objectListing = client.listNextBatchOfObjects(objectListing); } else { break; } } final VersionListing list = client.listVersions( new ListVersionsRequest().withBucketName(S3_BUCKET_NAME)); for (S3VersionSummary s : list.getVersionSummaries()) { client.deleteVersion(S3_BUCKET_NAME, s.getKey(), s.getVersionId()); } client.deleteBucket(S3_BUCKET_NAME); } }

Amazon适用于 Java 的 SDK 2.x 示例:使用亚马逊 S3 管理大型亚马逊 SQS 消息

以下Amazon适用于 Java 的 SDK 2.x 示例创建了一个名称随机的 Amazon S3 存储桶,并添加了一条生命周期规则,用于在 14 天后永久删除对象。它还会创建一个名为 MyQueue 的队列并向该队列发送一条存储在 S3 存储桶中且大小超过 256 KB 的随机消息。最后,代码会检索该消息,返回该消息的相关信息,并删除该消息、队列和存储桶。

/* * Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * Licensed under the Apache License, Version 2.0 (the "License"). * You may not use this file except in compliance with the License. * A copy of the License is located at * * https://aws.amazon.com/apache2.0 * * or in the "license" file accompanying this file. This file is distributed * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either * express or implied. See the License for the specific language governing * permissions and limitations under the License. * */ import com.amazon.sqs.javamessaging.AmazonSQSExtendedClient; import com.amazon.sqs.javamessaging.ExtendedClientConfiguration; import org.joda.time.DateTime; import org.joda.time.format.DateTimeFormat; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.BucketLifecycleConfiguration; import software.amazon.awssdk.services.s3.model.CreateBucketRequest; import software.amazon.awssdk.services.s3.model.DeleteBucketRequest; import software.amazon.awssdk.services.s3.model.DeleteObjectRequest; import software.amazon.awssdk.services.s3.model.ExpirationStatus; import software.amazon.awssdk.services.s3.model.LifecycleExpiration; import software.amazon.awssdk.services.s3.model.LifecycleRule; import software.amazon.awssdk.services.s3.model.LifecycleRuleFilter; import software.amazon.awssdk.services.s3.model.ListObjectVersionsRequest; import software.amazon.awssdk.services.s3.model.ListObjectVersionsResponse; import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; import software.amazon.awssdk.services.s3.model.ListObjectsV2Response; import software.amazon.awssdk.services.s3.model.PutBucketLifecycleConfigurationRequest; import software.amazon.awssdk.services.sqs.SqsClient; import software.amazon.awssdk.services.sqs.model.CreateQueueRequest; import software.amazon.awssdk.services.sqs.model.CreateQueueResponse; import software.amazon.awssdk.services.sqs.model.DeleteMessageRequest; import software.amazon.awssdk.services.sqs.model.DeleteQueueRequest; import software.amazon.awssdk.services.sqs.model.Message; import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest; import software.amazon.awssdk.services.sqs.model.ReceiveMessageResponse; import software.amazon.awssdk.services.sqs.model.SendMessageRequest; import java.util.Arrays; import java.util.List; import java.util.UUID; /** * Examples of using Amazon SQS Extended Client Library for Java 2.x * */ public class SqsExtendedClientExamples { // Create an Amazon S3 bucket with a random name. private final static String S3_BUCKET_NAME = UUID.randomUUID() + "-" + DateTimeFormat.forPattern("yyMMdd-hhmmss").print(new DateTime()); public static void main(String[] args) { /* * Create a new instance of the builder with all defaults (credentials * and region) set automatically. For more information, see * Creating Service Clients in the AWS SDK for Java Developer Guide. */ final S3Client s3 = S3Client.create(); /* * Set the Amazon S3 bucket name, and then set a lifecycle rule on the * bucket to permanently delete objects 14 days after each object's * creation date. */ final LifecycleRule lifeCycleRule = LifecycleRule.builder() .expiration(LifecycleExpiration.builder().days(14).build()) .filter(LifecycleRuleFilter.builder().prefix("").build()) .status(ExpirationStatus.ENABLED) .build(); final BucketLifecycleConfiguration lifecycleConfig = BucketLifecycleConfiguration.builder() .rules(lifeCycleRule) .build(); // Create the bucket and configure it s3.createBucket(CreateBucketRequest.builder().bucket(S3_BUCKET_NAME).build()); s3.putBucketLifecycleConfiguration(PutBucketLifecycleConfigurationRequest.builder() .bucket(S3_BUCKET_NAME) .lifecycleConfiguration(lifecycleConfig) .build()); System.out.println("Bucket created and configured."); // Set the Amazon SQS extended client configuration with large payload support enabled final ExtendedClientConfiguration extendedClientConfig = new ExtendedClientConfiguration().withPayloadSupportEnabled(s3, S3_BUCKET_NAME); final SqsClient sqsExtended = new AmazonSQSExtendedClient(SqsClient.builder().build(), extendedClientConfig); // Create a long string of characters for the message object int stringLength = 300000; char[] chars = new char[stringLength]; Arrays.fill(chars, 'x'); final String myLongString = new String(chars); // Create a message queue for this example final String queueName = "MyQueue-" + UUID.randomUUID(); final CreateQueueResponse createQueueResponse = sqsExtended.createQueue(CreateQueueRequest.builder().queueName(queueName).build()); final String myQueueUrl = createQueueResponse.queueUrl(); System.out.println("Queue created."); // Send the message final SendMessageRequest sendMessageRequest = SendMessageRequest.builder() .queueUrl(myQueueUrl) .messageBody(myLongString) .build(); sqsExtended.sendMessage(sendMessageRequest); System.out.println("Sent the message."); // Receive the message final ReceiveMessageResponse receiveMessageResponse = sqsExtended.receiveMessage(ReceiveMessageRequest.builder().queueUrl(myQueueUrl).build()); List<Message> messages = receiveMessageResponse.messages(); // Print information about the message for (Message message : messages) { System.out.println("\nMessage received."); System.out.println(" ID: " + message.messageId()); System.out.println(" Receipt handle: " + message.receiptHandle()); System.out.println(" Message body (first 5 characters): " + message.body().substring(0, 5)); } // Delete the message, the queue, and the bucket final String messageReceiptHandle = messages.get(0).receiptHandle(); sqsExtended.deleteMessage(DeleteMessageRequest.builder().queueUrl(myQueueUrl).receiptHandle(messageReceiptHandle).build()); System.out.println("Deleted the message."); sqsExtended.deleteQueue(DeleteQueueRequest.builder().queueUrl(myQueueUrl).build()); System.out.println("Deleted the queue."); deleteBucketAndAllContents(s3); System.out.println("Deleted the bucket."); } private static void deleteBucketAndAllContents(S3Client client) { ListObjectsV2Response listObjectsResponse = client.listObjectsV2(ListObjectsV2Request.builder().bucket(S3_BUCKET_NAME).build()); listObjectsResponse.contents().forEach(object -> { client.deleteObject(DeleteObjectRequest.builder().bucket(S3_BUCKET_NAME).key(object.key()).build()); }); ListObjectVersionsResponse listVersionsResponse = client.listObjectVersions(ListObjectVersionsRequest.builder().bucket(S3_BUCKET_NAME).build()); listVersionsResponse.versions().forEach(version -> { client.deleteObject(DeleteObjectRequest.builder().bucket(S3_BUCKET_NAME).key(version.key()).versionId(version.versionId()).build()); }); client.deleteBucket(DeleteBucketRequest.builder().bucket(S3_BUCKET_NAME).build()); } }

您可以使用 Apache Maven 为您的 Java 项目配置和构建 Amazon SQS 扩展客户端,也可以自己构建 SDK。从 SDK 中指定您在应用程序中使用的各个模块。

<properties> <aws-java-sdk.version>2.20.153</aws-java-sdk.version> </properties> <dependencies> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>sqs</artifactId> <version>${aws-java-sdk.version}</version> </dependency> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>s3</artifactId> <version>${aws-java-sdk.version}</version> </dependency> <dependency> <groupId>com.amazonaws</groupId> <artifactId>amazon-sqs-java-extended-client-lib</artifactId> <version>2.0.4</version> </dependency> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.12.6</version> </dependency> </dependencies>