Amazon S3 存储桶生命周期操作脚本示例 - AWS Command Line Interface
AWS 文档中描述的 AWS 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅中国的 AWS 服务入门

Amazon S3 存储桶生命周期操作脚本示例

本主题介绍使用 AWS Command Line Interface (AWS CLI) 的 Amazon S3 存储桶生命周期操作 bash 脚本示例。此脚本示例使用 aws s3api 命令集。Shell 脚本是专用于在命令行界面中运行的程序。

在您开始之前

您需要先满足以下条件,才能运行下文中的任何示例代码。

  • 安装 AWS CLI,请参阅安装 AWS CLI了解更多信息。

  • 配置 AWS CLI,请参阅配置基础知识了解更多信息。您使用的配置文件必须具有允许示例代码执行 AWS 操作的权限。

  • 作为 AWS 的最佳实践,请授予此代码最低的权限,或者仅授予它执行任务所需的权限。有关更多信息,请参阅 AWS Identity and Access Management (IAM) 用户指南中的授予最低权限

  • 此代码尚未在所有 AWS 区域中进行测试。有些 AWS 服务仅在特定区域中提供。有关更多信息,请参阅 AWS 一般参考指南中的服务终端节点和配额

  • 运行此代码可能会导致您的 AWS 账户产生相关费用。您有责任确保在使用完由此脚本创建的任何资源后删除这些资源。

Amazon S3 服务使用以下术语:

  • 存储桶 - 顶层 Amazon S3 文件夹。

  • 前缀 - 存储桶中的 Amazon S3 文件夹。

  • 对象 - Amazon S3 存储桶中托管的任何项目。

关于此示例

此示例说明如何使用 shell 脚本文件中的一组函数与一些基本的 Amazon S3 操作进行交互。这些函数包含在名为 bucket-operations.sh 的 shell 脚本文件中。您可以在另一个文件中调用这些函数。每个脚本文件都包含了介绍每个函数的注释。

要查看每个步骤的中间结果,请使用 -i 参数运行脚本。您可以使用 Amazon S3 控制台查看存储桶或其内容的当前状态。仅当您在系统提示符处按 Enter 后,脚本才会继续执行下一步。

有关完整示例和可下载的脚本文件,请参阅 GitHub 上的 AWS 代码示例存储库中的 Amazon S3 存储桶生命周期操作

文件

此示例包含以下文件:

bucket-operations.sh

此主脚本文件可以从另一个文件中调取。它包含了执行以下任务的函数:

  • 创建存储桶并验证它是否存在

  • 将文件从本地计算机复制到存储桶

  • 将文件从一个存储桶位置复制到另一个存储桶位置

  • 列出存储桶中的内容

  • 从存储桶中删除文件

  • 删除存储桶

source ./awsdocs_general.sh ############################################################################### # function bucket_exists # # This function checks to see if the specified bucket already exists. # # Parameters: # $1 - The name of the bucket to check # # Returns: # 0 if the bucket already exists # 1 if the bucket doesn't exist ############################################################################### function bucket_exists { be_bucketname=$1 # Check whether the bucket already exists. # We suppress all output - we're interested only in the return code. aws s3api head-bucket \ --bucket $be_bucketname \ >/dev/null 2>&1 if [[ ${?} -eq 0 ]]; then return 0 # 0 in Bash script means true. else return 1 # 1 in Bash script means false. fi } ############################################################################### # function create-bucket # # This function creates the specified bucket in the specified AWS Region, unless # it already exists. # # Parameters: # -b bucket_name -- The name of the bucket to create # -r region_code -- The code for an AWS Region in which to # create the bucket # # Returns: # The URL of the bucket that was created. # And: # 0 if successful # 1 if it fails ############################################################################### function create_bucket { local BUCKET_NAME REGION_CODE RESPONSE local OPTION OPTIND OPTARG # Required to use getopts command in a function function usage { echo "function create_bucket" echo "Creates an Amazon S3 bucket. You must supply both of the following parameters:" echo " -b bucket_name The name of the bucket. It must be globally unique." echo " -r region_code The code for an AWS Region in which the bucket is created." echo "" } # Retrieve the calling parameters while getopts "b:r:" OPTION; do case "${OPTION}" in b) BUCKET_NAME="${OPTARG}";; r) REGION_CODE="${OPTARG}";; h) usage; return 0;; \?) echo "Invalid parameter"; usage; return 1;; esac done if [[ -z "$BUCKET_NAME" ]]; then errecho "ERROR: You must provide a bucket name with the -b parameter." usage return 1 fi if [[ -z "$REGION_CODE" ]]; then errecho "ERROR: You must provide an AWS Region code with the -r parameter." usage return 1 fi iecho "Parameters:\n" iecho " Bucket name: $BUCKET_NAME" iecho " Region code: $REGION_CODE" iecho "" # If the bucket already exists, we don't want to try to create it. if (bucket_exists $BUCKET_NAME); then errecho "ERROR: A bucket with that name already exists. Try again." return 1 fi # The bucket doesn't exist, so try to create it. RESPONSE=$(aws s3api create-bucket \ --bucket $BUCKET_NAME \ --create-bucket-configuration LocationConstraint=$REGION_CODE) if [[ ${?} -ne 0 ]]; then errecho "ERROR: AWS reports create-bucket operation failed.\n$RESPONSE" return 1 fi } ############################################################################### # function copy_file_to_bucket # # This function creates a file in the specified bucket. # # Parameters: # -b bucket_name$1 - The name of the bucket to copy the file to # $2 - The path and file name of the local file to copy to the bucket # $3 - The key (name) to call the copy of the file in the bucket # # Returns: # 0 if successful # 1 if it fails ############################################################################### function copy_file_to_bucket { cftb_bucketname=$1 cftb_sourcefile=$2 cftb_destfilename=$3 local RESPONSE RESPONSE=$(aws s3api put-object \ --bucket $cftb_bucketname \ --body $cftb_sourcefile \ --key $cftb_destfilename) if [[ ${?} -ne 0 ]]; then errecho "ERROR: AWS reports put-object operation failed.\n$RESPONSE" return 1 fi } ############################################################################### # function copy_item_in_bucket # # This function creates a copy of the specified file in the same bucket. # # Parameters: # $1 - The name of the bucket to copy the file from and to # $2 - The key of the source file to copy # $3 - The key of the destination file # # Returns: # 0 if successful # 1 if it fails ############################################################################### function copy_item_in_bucket { ciib_bucketname=$1 ciib_sourcefile=$2 ciib_destfile=$3 local RESPONSE RESPONSE=$(aws s3api copy-object \ --bucket $ciib_bucketname \ --copy-source $ciib_bucketname/$ciib_sourcefile \ --key $ciib_destfile) if [[ $? -ne 0 ]]; then errecho "ERROR: AWS reports s3api copy-object operation failed.\n$RESPONSE" return 1 fi } ############################################################################### # function list_items_in_bucket # # This function displays a list of the files in the bucket with each file's # size. The function uses the --query parameter to retrieve only the Key and # Size fields from the Contents collection. # # Parameters: # $1 - The name of the bucket # # Returns: # The list of files in text format # And: # 0 if successful # 1 if it fails ############################################################################### function list_items_in_bucket { liib_bucketname=$1 local RESPONSE RESPONSE=$(aws s3api list-objects \ --bucket $liib_bucketname \ --output text \ --query 'Contents[].{Key: Key, Size: Size}' ) if [[ ${?} -eq 0 ]]; then echo "$RESPONSE" else errecho "ERROR: AWS reports s3api list-objects operation failed.\n$RESPONSE" return 1 fi } ############################################################################### # function delete_item_in_bucket # # This function deletes the specified file from the specified bucket. # # Parameters: # $1 - The name of the bucket # $2 - The key (file name) in the bucket to delete # Returns: # 0 if successful # 1 if it fails ############################################################################### function delete_item_in_bucket { diib_bucketname=$1 diib_key=$2 local RESPONSE RESPONSE=$(aws s3api delete-object \ --bucket $diib_bucketname \ --key $diib_key) if [[ $? -ne 0 ]]; then errecho "ERROR: AWS reports s3api delete-object operation failed.\n$RESPONSE" return 1 fi } ############################################################################### # function delete_bucket # # This function deletes the specified bucket. # # Parameters: # $1 - The name of the bucket # Returns: # 0 if successful # 1 if it fails ############################################################################### function delete_bucket { db_bucketname=$1 local RESPONSE RESPONSE=$(aws s3api delete-bucket \ --bucket $db_bucketname) if [[ $? -ne 0 ]]; then errecho "ERROR: AWS reports s3api delete-bucket failed.\n$RESPONSE" return 1 fi }
test-bucket-operations.sh

shell 脚本文件 test-bucket-operations.sh 演示了如何调用这些函数,即调取 bucket-operations.sh 文件然后调用其中的每个函数。调用函数后,该测试脚本会删除它创建的所有资源。

source ./awsdocs_general.sh source ./bucket_operations.sh function usage { echo "This script tests Amazon S3 bucket operations in the AWS CLI." echo "It creates a randomly named bucket, copies files to it, then" echo "deletes the files and the bucket." echo "" echo "To pause the script between steps so you can see the results in the" echo "AWS Management Console, include the parameter -i." echo "" echo "IMPORTANT: Running this script creates resources in your Amazon" echo " account that can incur charges. It is your responsibility to" echo " ensure that no resources are left in your account after the script" echo " completes. If an error occurs during the operation of the script," echo " then resources can remain that you might need to delete manaully." } # Set default values. INTERACTIVE=false VERBOSE=false # Retrieve the calling parameters while getopts "ivh" OPTION; do case "${OPTION}" in i) INTERACTIVE=true;VERBOSE=true; iecho;; v) VERBOSE=true;; h) usage; return 0;; \?) echo "Invalid parameter"; usage; return 1;; esac done if [ "$INTERACTIVE" == "true" ]; then iecho "Tests running in interactive mode."; fi if [ "$VERBOSE" == "true" ]; then iecho "Tests running in verbose mode."; fi iecho "***************SETUP STEPS******************" BUCKETNAME=$(generate_random_name s3test) REGION="us-west-2" FILENAME1=$(generate_random_name s3testfile) FILENAME2=$(generate_random_name s3testfile) iecho "BUCKETNAME=$BUCKETNAME" iecho "REGION=$REGION" iecho "FILENAME1=$FILENAME1" iecho "FILENAME2=$FILENAME2" iecho "**************END OF STEPS******************" run_test "1. Creating bucket with missing bucket_name" \ "create_bucket -r $REGION" \ 1 \ "ERROR: You must provide a bucket name" \ run_test "2. Creating bucket with missing region_name" \ "create_bucket -b $BUCKETNAME" \ 1 \ "ERROR: You must provide an AWS Region code" run_test "3. Creating bucket with valid parameters" \ "create_bucket -r $REGION -b $BUCKETNAME" \ 0 run_test "4. Creating bucket with duplicate name and region" \ "create_bucket -r $REGION -b $BUCKETNAME" \ 1 \ "ERROR: A bucket with that name already exists" run_test "5. Copying local file (copy of this script) to bucket" \ "copy_file_to_bucket $BUCKETNAME ./$0 $FILENAME1" \ 0 run_test "6. Duplicating existing file in bucket" \ "copy_item_in_bucket $BUCKETNAME $FILENAME1 $FILENAME2" \ 0 run_test "7. Listing contents of bucket" \ "list_items_in_bucket $BUCKETNAME" \ 0 run_test "8. Deleting first file from bucket" \ "delete_item_in_bucket $BUCKETNAME $FILENAME1" \ 0 run_test "9. Deleting second file from bucket" \ "delete_item_in_bucket $BUCKETNAME $FILENAME2" \ 0 run_test "10. Deleting bucket" \ "delete_bucket $BUCKETNAME" \ 0 echo "Tests completed successfully."
awsdocs-general.sh

脚本文件 awsdocs-general.sh 中包含了在 AWS CLI 的高级代码示例中使用的通用函数。

# Set global defaults: VERBOSE=false ############################################################################### # function run_test # # This function is used to perform a command and compare its output to both # the expected error code and the expected output string. If there isn't a # match, then the function invokes the test_failed function. ############################################################################### function run_test { local DESCRIPTION COMMAND EXPECTED_ERR_CODE EXPECTED_OUTPUT RESPONSE DESCRIPTION="$1" COMMAND="$2" EXPECTED_ERR_CODE="$3" if [[ -z "$4" ]]; then EXPECTED_OUTPUT="$4"; else EXPECTED_OUTPUT=""; fi iecho -n "Running test: $DESCRIPTION..." RESPONSE="$($COMMAND)" ERR="${?}" # Check to see if we got the expected error code. if [[ "$EXPECTED_ERR_CODE" -ne "$ERR" ]]; then test_failed "The test \"$DESCRIPTION\" returned an unexpected error code: $ERR" fi #now check the error message, if we provided other than "". if [[ -n "$EXPECTED_OUTPUT" ]]; then MATCH=$(echo "$RESPONSE" | grep "$EXPECTED_OUTPUT") # If there was no match (it's an empty string), then fail. if [[ -z "$MATCH" ]]; then test_failed "The test \"$DESCRIPTION\" returned an unexpected output: $RESPONSE" fi fi iecho "OK" ipause } ############################################################################### # function test_failed # # This function is used to terminate a failed test and to warn the customer # about possible undeleted resources that could incur costs to their account. ############################################################################### function test_failed { errecho "" errecho "===TEST FAILED===" errecho "$@" errecho "" errecho " One or more of the tests failed to complete successfully. This means that any" errecho " tests after the one that failed test didn't run and might have left resources" errecho " still active in your account." errecho "" errecho "IMPORTANT:" errecho " Resources created by this script can incur charges to your AWS account. If the" errecho " script did not complete successfully, then you must review and manually delete" errecho " any resources created by this script that were not automatically removed." errecho "" exit 1 } ############################################################################### # function errecho # # This function outputs everything sent to it to STDERR (standard error output). ############################################################################### function errecho { printf "%s\n" "$*" 2>&1 } ############################################################################### # function iecho # # This function enables the script to display the specified text only if # the global variable $VERBOSE is set to true. ############################################################################### function iecho { if [[ $VERBOSE == true ]]; then echo "$@" fi } ############################################################################### # function ipause # # This function enables the script to pause after each command if interactive # mode is set (by including -i on the script invocation command). ############################################################################### function ipause { if [[ $INTERACTIVE == true ]]; then read -r -p "Press ENTER to continue..." fi } # Initialize the shell's RANDOM variable RANDOM=$$ ############################################################################### # function generate_random_name # # This function generates a random file name with using the specified root # followed by 4 groups that each have 4 digits. # The default root name is "test" function generate_random_name { ROOTNAME="test" if [[ -n $1 ]]; then ROOTNAME=$1 fi # Initialize the filename variable FILENAME="$ROOTNAME" # Configure random number generator to issue numbers between 1000 and 9999, inclusive DIFF=$((9999-1000+1)) for _ in {1..4} do rnd=$(($((RANDOM%DIFF))+X)) # make sure that the number is 4 digits long while [ "${#rnd}" -lt 4 ]; do rnd="0$rnd"; done FILENAME+="-$rnd" done echo $FILENAME }

参考

AWS CLI 参考:

其他参考资料: