Migrating Amazon Glue for Spark jobs to Amazon Glue version 5.1

Apache Spark update from 3.5.4 in Amazon Glue 5.0 to 3.5.6 in Amazon Glue 5.1.

Open Table Formats (OTF) updated to Hudi 1.0.2, Iceberg 1.10.0, and Delta Lake 3.3.2

Iceberg Materialized Views - Create and manage Iceberg Materialized Views (MV). For more information, see blog post

Iceberg format version 3.0 - Extends data types and existing metadata structures to add new capabilities. For more information, see the Iceberg Table Spec.

Hudi Full Table Access - Full Table Access (FTA) control for Apache Hudi in Apache Spark based on your policies defined in . This feature enables read and write operations from your Amazon Glue ETL jobs on registered tables when the job role has full table access.

Spark native fine-grained access control (FGAC) support using - DDL/DML operations (like CREATE, ALTER, DELETE, DROP) with fine grained access control for Apache Hive, Apache Iceberg and Delta Lake tables registered in .

Audit context for Spark jobs - Audit context for Amazon Glue ETL jobs will be available for Amazon Glue and Amazon Lake Formation API calls in the Amazon CloudTrail logs

Limited support for view SQL clause for creation of materialized views, query rewrite and incremental refresh. More details can be found in the Iceberg Materialized Views feature documentation page

Hudi FTA writes require using HoodieCredentialedHadoopStorage for credential vending during job execution. Set the following configuration when running Hudi jobs:

hoodie.storage.class=org.apache.spark.sql.hudi.storage.HoodieCredentialedHadoopStorage

Hudi FTA write support works only with the default Hudi configurations. Custom or non-default Hudi settings may not be fully supported and could result in unexpected behavior. Clustering for Hudi Merge-On-Read (MOR) tables is also not supported under FTA write mode.

S3A filesystem has replaced EMRFS as the default S3 connector. For information on how to migrate, see Migrating from Amazon Glue 5.0 to Amazon Glue 5.1.

In Amazon Glue Studio, choose Glue 5.1 - Supports Spark 3.5.6, Scala 2, Python 3 in Glue version.

In the API, choose 5.1 in the GlueVersion parameter in the UpdateJob API operation.

In the console, choose Spark 3.5.6, Python 3 (Glue Version 5.1) or Spark 3.5.6, Scala 2 (Glue Version 5.1) in Glue version.

In Amazon Glue Studio, choose Glue 5.1 - Supports Spark 3.5.6, Scala 2, Python 3 in Glue version.

In the API, choose 5.1 in the GlueVersion parameter in the CreateJob API operation.

[Python] Update boto references from 1.34 to 1.40.

In Amazon Glue 5.1, S3A filesystem has replaced EMRFS as the default S3 connector. If both spark.hadoop.fs.s3a.endpoint and spark.hadoop.fs.s3a.endpoint.region are not set, the default region used by S3A is us-east-2. This can cause issues, such as S3 upload timeout errors, especially for VPC jobs. To mitigate the issues caused by this change, set the spark.hadoop.fs.s3a.endpoint.region Spark configuration when using the S3A file system in Amazon Glue 5.1.

To continue using EMRFS instead of S3A, set the following spark configurations:

    --conf spark.hadoop.fs.s3.impl=com.amazon.ws.emr.hadoop.fs.EmrFileSystem
    --conf spark.hadoop.fs.s3n.impl=com.amazon.ws.emr.hadoop.fs.EmrFileSystem
    --conf spark.hadoop.fs.AbstractFileSystem.s3.impl=org.apache.hadoop.fs.s3.EMRFSDelegate
                

Migration Guide: Spark Core

Migration Guide: SQL, Datasets and DataFrame

Migration Guide: Structured Streaming

Upgrading PySpark

For migration steps related to Amazon Glue 4.0 to Amazon Glue 5.0, see Migrating from Amazon Glue 4.0 to Amazon Glue 5.0.

For migration steps related to Amazon Glue 3.0 to Amazon Glue 5.0, see Migrating from Amazon Glue 3.0 to Amazon Glue 5.0.

For migration steps related to Amazon Glue 2.0 to Amazon Glue 5.0 and a list of migration differences between Amazon Glue version 2.0 and 4.0, see Migrating from Amazon Glue 2.0 to Amazon Glue 5.0.

Appendix B: JDBC driver upgrades

Appendix C: Connector upgrades

Appendix D: Open table format upgrades

Support FTA read and write access on Lake Formation registered tables.

Support Iceberg format version 3. The following features are supported:

Support Spark-native FGAC writes on registered tables.

Athena SQL compatibility - Cannot read Iceberg V3 tables created by EMR Spark due to error: GENERIC_INTERNAL_ERROR: Cannot read unsupported version 3

Support FTA read and write access on Lake Formation registered tables.