Creating metadata table configurations
To generate and store Amazon S3 Metadata in a fully managed Apache Iceberg metadata table, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata table to reflect the latest changes to your data as long as the configuration is active on the bucket. Additionally, Amazon S3 continuously optimizes your metadata table to help reduce storage costs and improve analytics query performance.
Metadata tables have the following Amazon Resource Name (ARN) format:
arn:aws-cn:s3tables:
region-code
:account-id
:bucket/table-bucket-name
/table/metadata_table_name
Amazon S3 fully managed metadata tables are stored in the aws_s3_metadata
namespace in your table bucket. For more information about namespaces in table buckets, see
Table namespaces.
You can create a metadata table configuration by using the Amazon S3 console, the Amazon Command Line Interface (Amazon CLI), the Amazon SDKs, or the Amazon S3 REST API.
Prerequisites
To create a metadata table configuration, you must first do the following:
-
Make sure that you have the necessary Amazon Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see Setting up permissions for configuring metadata tables.
-
Create an S3 table bucket to store your metadata table in. This table bucket must be in the same Amazon Web Services Region and Amazon Web Services account as your general purpose bucket. For more information about creating table buckets, see Creating table buckets. If you're using the Amazon S3 console to create your configuration, you can do this step as part of that process.
-
Integrate your table bucket with Amazon Glue Data Catalog so that you can directly query your metadata tables with query engines such as Amazon Athena, Amazon EMR, Amazon Redshift, Apache Spark, Apache Trino, and any other application that supports the Apache Iceberg format. For more information, see Querying metadata tables with Amazon analytics services.
Create a metadata table configuration
To create a metadata table configuration
Before you create a metadata table configuration, make sure that you've reviewed and met the prerequisites and that you've reviewed Metadata table limitations and restrictions.
Sign in to the Amazon Web Services Management Console and open the Amazon S3 console at https://console.amazonaws.cn/s3/
. -
In the left navigation pane, choose General purpose buckets.
-
Choose the general purpose bucket that you want to create a metadata table configuration for.
Note
Make sure that this general purpose bucket is an Amazon Web Services Region where table buckets are available. Table buckets are available only in the US East (N. Virginia), US East (Ohio), and US West (Oregon) Regions.
-
On the buckets' details page, choose the Metadata tab.
-
On the Metadata tab, choose Create metadata configuration.
-
On the Create metadata configuration page, under Destination table bucket, specify a table bucket to store your metadata table in. The table bucket must be in the same Amazon Web Services account and Amazon Web Services Region as your general purpose bucket.
If you don't already have a table bucket, choose Create table bucket. Table bucket names must be 3 to 63 characters and unique within your Amazon Web Services account in the Amazon Web Services Region that you've chosen. Valid characters are a-z, 0-9, and hyphens (-). For more information about creating a table bucket, see Creating table buckets.
When you create your table bucket, make sure that you integrate it with Amazon Glue Data Catalog. For more information, see Querying metadata tables with Amazon analytics services.
-
For Metadata table name, specify the name that you want your table to have. The metadata table name must be between 1 and 255 characters and unique within the
aws_s3_metadata
namespace in your table bucket. Valid characters are lowercase letters, numbers, and underscores (_
). -
Choose Create metadata table configuration.
If your metadata table configuration was successful, the ARN for your metadata table is displayed on the Metadata tab, along with the specified table bucket and metadata table name.
To monitor updates to your metadata table configuration, you can use Amazon CloudTrail. For more information, see Amazon S3 bucket-level actions that are tracked by CloudTrail logging.
To run the following commands, you must have the Amazon CLI installed and configured. If you don’t have the Amazon CLI installed, see Install or update to the latest version of the Amazon CLI in the Amazon Command Line Interface User Guide.
Alternatively, you can run Amazon CLI commands from the console by using Amazon CloudShell. Amazon CloudShell is a browser-based, pre-authenticated shell that you can launch directly from the Amazon Web Services Management Console. For more information, see What is CloudShell? and Getting started with Amazon CloudShell in the Amazon CloudShell User Guide.
To create a metadata table configuration by using the Amazon CLI
Before you create a metadata table configuration, make sure that you've reviewed and met the prerequisites and that you've reviewed Metadata table limitations and restrictions.
To use the following example commands, replace the
with your own information. user input
placeholders
-
If you don't already have a table bucket, use the following command to create a table bucket to store your metadata table in. Make sure the table bucket is in the same Amazon Web Services Region as the general purpose bucket that you want to create a metadata table configuration for.
aws s3tables create-table-bucket --name
amzn-s3-demo-bucket
--regionus-east-2
-
To verify that your table bucket has been created, use the following command:
aws s3tables list-table-buckets --region
us-east-2
-
Create a JSON file that contains your metadata table configuration, and save it (for example,
metadata-config.json
). The following is a sample configuration.Table bucket names must be 3 to 63 characters and unique within your Amazon Web Services account in the Amazon Web Services Region that you've chosen. Valid characters are a-z, 0-9, and hyphens (-). For more information about creating a table bucket, see Creating table buckets.
The metadata table name must be between 1 and 255 characters and unique within the
aws_s3_metadata
namespace in your table bucket. Valid characters are lowercase letters, numbers, and underscores (_
).{ "S3TablesDestination": { "TableBucketArn": "arn:aws-cn:s3tables:
us-east-2
:111122223333
:bucket/amzn-s3-demo-bucket
", "TableName": "test_metadata_table
" } } -
Use the following command to apply the metadata table configuration to your general purpose bucket (for example,
):amzn-s3-demo-source-bucket
aws s3api create-bucket-metadata-table-configuration \ --bucket
\ --metadata-table-configuration file://./amzn-s3-demo-source-bucket
metadata-config
.json \ --regionus-east-2
-
To verify that the configuration was created, use the following command:
aws s3api get-bucket-metadata-table-configuration \ --bucket
\ --regionamzn-s3-demo-source-bucket
us-east-2
To monitor updates to your metadata table configuration, you can use Amazon CloudTrail. For more information, see Amazon S3 bucket-level actions that are tracked by CloudTrail logging.
You can send REST requests to create a metadata table configuration. For more information, see CreateBucketMetadataTableConfiguration in the Amazon S3 API Reference.
You can use the Amazon SDKs to create a metadata table configuration in Amazon S3. For information, see the list of supported SDKs in the Amazon S3 API Reference.