Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Connecting to the Data Catalog using Amazon Glue Iceberg REST endpoint

Amazon Glue's Iceberg REST endpoint supports API operations specified in the Apache Iceberg REST specification. Using an Iceberg REST client, you can connect your application running on an analytics engine to the REST catalog hosted in the Data Catalog.

The endpoint supports both Apache Iceberg table specifications - v1 and v2, defaulting to v2. When using the Iceberg table v1 specification, you must specify v1 in the API call. Using the API operation, you can access Iceberg tables stored in both Amazon S3 object storage and Amazon S3 Table storage.

Endpoint configuration

You can access the Amazon Glue Iceberg REST catalog using the service endpoint. Refer to the Amazon Glue service endpoints reference guide for the region-specific endpoint. For example, when connecting to Amazon Glue in the us-east-1 Region, you need to configure the endpoint URI property as follows:

Endpoint : https://glue.us-east-1.amazonaws.com/iceberg

Additional configuration properties – When using Iceberg client to connect an analytics engine like Spark to the service endpoint, you are required to specify the following application configuration properties:

catalog_name = "mydatacatalog"
aws_account_id = "123456789012"
aws_region = "us-east-1"
spark = SparkSession.builder \
    ... \
    .config("spark.sql.defaultCatalog", catalog_name) \
    .config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") \
    .config(f"spark.sql.catalog.{catalog_name}.type", "rest") \
    .config(f"spark.sql.catalog.{catalog_name}.uri", "https://glue.{aws_region}.amazonaws.com/iceberg") \
    .config(f"spark.sql.catalog.{catalog_name}.warehouse", "{aws_account_id}") \
    .config(f"spark.sql.catalog.{catalog_name}.rest.sigv4-enabled", "true") \
    .config(f"spark.sql.catalog.{catalog_name}.rest.signing-name", "glue") \    
    .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .getOrCreate()
     

Amazon Glue Iceberg endpoint https://glue.us-east-1.amazonaws.com/iceberg supports supports the following Iceberg REST APIs:

Authentication

The Amazon Glue Iceberg REST endpoint supports two authentication methods. Choose the method that matches how your analytics engine or integration connects to the catalog.

Authenticating with OAuth 2.0 bearer tokens

External clients obtain a short-lived OAuth access token from Amazon Sign-In and present it to the Amazon Glue Iceberg REST endpoint. Amazon Glue validates the token and enforces Amazon Lake Formation permissions on every request. For the LoadTable operation, the Amazon S3 credentials that Amazon Glue vends are bounded by the assumed IAM role and the Lake Formation grants on that role.

This integration uses the OAuth 2.0 Token Exchange Grant (RFC 8693). An integration partner presents an end user's JSON Web Token (JWT) to Amazon Sign-In and exchanges it for a short-lived OAuth access token scoped to the Amazon Glue service. The partner then presents that token to the Amazon Glue Iceberg REST endpoint to read your tables.

For the complete setup and token exchange details, see Configure OAuth access to Amazon Glue Iceberg REST Catalog in the Amazon Sign-In User Guide.

Prefix and catalog path parameters

Iceberg REST catalog APIs have a free-form prefix in their request URLs. For example, the ListNamespaces API call uses the GET/v1/{prefix}/namespaces URL format. Amazon Glue prefix always follows the /catalogs/{catalog} structure to ensure that the REST path aligns the Amazon Glue multi-catalog hierarchy. The {catalog} path parameter can be derived based on the following rules:

This catalog ID to prefix mapping is required only when you directly call the REST APIs. When you are working with the Amazon Glue Iceberg REST catalog APIs through an engine, you need to specify the Amazon Glue catalog ID in the warehouse parameter for your Iceberg REST catalog API setting or in the glue.id parameter for your Amazon Glue extensions API setting. For example, see how you can use it with EMR Spark in Use an Iceberg cluster with Spark.

Namespace path parameter

Namespaces in Iceberg REST catalog APIs path can have multiple levels. However, Amazon Glue only supports single-level namespaces. To access a namespace in a multi-level catalog hierarchy, you can connect to a multi-level catalog above the namespace to reference the namespace. This allows any query engine that supports the 3-part notation of catalog.namespace.table to access objects in Amazon Glue’s multi-level catalog hierarchy without compatibility issues compared to using the multi-level namespace.