Migrating Amazon Glue for Spark jobs to Amazon Glue version 6.0

Apache Spark upgrade from 3.5.6 in Amazon Glue 5.1 to 4.1.1 in Amazon Glue 6.0.

Scala upgrade from 2.12.18 to 2.13.17.

Python upgrade from 3.11 to 3.13.

Open Table Formats (OTF) updated to Hudi 1.1.1, Iceberg 1.11.0, and Delta Lake 4.2.0.

Iceberg format version 3 — Extends data types and existing metadata structures to add new capabilities including:

Spark Declarative Pipelines (SDP) — A new declarative framework for defining end-to-end data pipelines using SQL or DataFrame API, with support for streaming tables and materialized views. For more information, see Spark Declarative Pipelines.

Spark Connect for Interactive Sessions — Enables thin-client connectivity to Amazon Glue Interactive Sessions through Spark Connect protocol, supporting remote development workflows.

Arrow-native Python UDFs/UDTFs — Improved performance for Python user-defined functions using Apache Arrow columnar format natively.

Customer-managed or service-generated Python virtual environment (--python-virtual-env) — You can build and provide your own Python venv that Amazon Glue attaches to Spark drivers and executors at runtime, providing full control over dependency management. When existing jobs are migrated to Amazon Glue 6.0, Amazon Glue automatically generates this virtual environment if required.

Streaming enhancements — Real-time mode for stateless streaming with millisecond-level latency. For more information, see Enabling real-time mode for streaming jobs.

Connector upgrades — Updated Amazon Redshift, MongoDB, Snowflake, and other connectors. See Appendix C: Connector upgrades for full version details.

ANSI mode is enabled by default in Spark 4.1. Operations that previously returned NULL on overflow (for example, integer arithmetic, cast operations) now throw exceptions. Set spark.sql.ansi.enabled=false to restore previous behavior.

Spark Declarative Pipelines (SDP) is a new feature with limited support for certain SQL constructs. Refer to the Spark Declarative Pipelines documentation for current limitations.

Iceberg v3 tables created in Amazon Glue 6.0 cannot be read by Athena SQL (error: Cannot read unsupported version 3). Use Iceberg v2 for cross-engine compatibility with Athena.

Python 3.13 removes several deprecated modules and changes behavior for some standard library functions. Review the Python 3.13 migration guide for compatibility.

Amazon SDK for Java 2.x compatibility with --user-jars-first — Amazon Glue 6.0 includes Amazon SDK for Java 2.x version 2.44.6. If you use the --user-jars-first job parameter with a custom JAR that bundles an older version of the Amazon SDK for Java 2.x, your job might fail with a java.lang.NoSuchFieldError or similar error. These failures occur when the SDK bundled in your custom JAR is missing classes, fields, or methods that the Amazon Glue runtime depends on. To avoid this issue, ensure that any custom JAR you supply with --user-jars-first uses Amazon SDK for Java 2.x version 2.44.6 or later.

EMRFS has been removed. S3A is the only S3 filesystem in Amazon Glue 6.0. The com.amazon.ws.emr.hadoop.fs.EmrFileSystem class is no longer available. Jobs using s3:// paths use S3A automatically. If you previously set EMRFS-specific configurations (for example, fs.s3.consistent.*), remove them.

Amazon SDK for Java v1 has been removed. Only Amazon SDK v2 (2.44.6) is available. Jobs importing com.amazonaws.services.* packages must migrate to software.amazon.awssdk.services.*.

Scala upgraded from 2.12 to 2.13. Custom JARs compiled against Scala 2.12 don't work. Recompile against Scala 2.13.17. Key changes: JavaConversions removed (use CollectionConverters), MutableList removed (use ListBuffer), parallel collections require separate import.

Spark 4.1 API changes:

CreateSession API validation — Additional validation on session parameters for Interactive Sessions. Invalid configurations that were previously silently accepted might now return errors.

getResolvedOptions behavior change — Argument prefix matching is disabled by default (allow_abbrev=False). Use the full argument name, or pass allow_abbrev=True to getResolvedOptions() to restore the old behavior.

In Amazon Glue Studio, choose Glue 6.0 - Supports Spark 4.1.1, Scala 2, Python 3 in Glue version.

In the API, choose 6.0 in the GlueVersion parameter in the UpdateJob API operation.

In the console, choose Spark 4.1.1, Python 3 (Glue Version 6.0) or Spark 4.1.1, Scala 2 (Glue Version 6.0) in Glue version.

In Amazon Glue Studio, choose Glue 6.0 - Supports Spark 4.1.1, Scala 2, Python 3 in Glue version.

In the API, choose 6.0 in the GlueVersion parameter in the CreateJob API operation.

[Scala] Recompile custom JARs against Scala 2.13.17. Replace JavaConversions with CollectionConverters.

[Python] Update code for Python 3.13 compatibility. Remove usage of deprecated modules (for example, imp, cgi, cgitb).

[Python] Update boto3 references from 1.40 to 1.42.

[Spark SQL] Review queries for ANSI mode impact. Add spark.sql.ansi.enabled=false if needed.

[SDK] Replace any Amazon SDK v1 imports (com.amazonaws.*) with SDK v2 (software.amazon.awssdk.*).

[S3] Remove EMRFS-specific configurations. S3A is now the default and only S3 connector.

[Dependencies] Update --extra-jars to versions compiled for Scala 2.13 and Spark 4.1.

S3 filesystem: EMRFS is no longer available. S3A is the sole S3 connector. If you previously set spark.hadoop.fs.s3a.endpoint.region, continue using it. If not set, ensure your VPC endpoint or network configuration allows S3A to resolve the correct region.

Scala binary compatibility: Amazon Glue 6.0 uses Scala 2.13. Any --extra-jars compiled with Scala 2.12 fail with NoSuchMethodError or ClassNotFoundException. Recompile all custom Scala/Java JARs.

Spark SQL behavior changes:

Python version: Python 3.13 is the runtime. The --additional-python-modules feature continues to work but is deprecated. Consider migrating to --python-virtual-env for full dependency control.

Amazon SDK: Only SDK v2 is available. If your scripts use boto3 (Python), no change is needed — boto3 continues to work. For Scala/Java jobs using the Amazon SDK directly, migrate to v2 APIs.

Migration Guide: Spark Core on the Apache Spark website

Migration Guide: SQL, Datasets and DataFrame on the Apache Spark website

Migration Guide: Structured Streaming on the Apache Spark website

Upgrading PySpark on the Apache Spark website

For migration steps related to Amazon Glue 5.0 to Amazon Glue 5.1, see Migrating from Amazon Glue 5.0 to Amazon Glue 5.1.

For migration steps related to Amazon Glue 4.0 to Amazon Glue 5.0, see Migrating from Amazon Glue 4.0 to Amazon Glue 5.0.

For migration steps related to Amazon Glue 3.0 to Amazon Glue 5.0, see Migrating from Amazon Glue 3.0 to Amazon Glue 5.0.

For migration steps related to Amazon Glue 2.0 to Amazon Glue 5.0, see Migrating from Amazon Glue 2.0 to Amazon Glue 5.0.

After completing the steps above, finish migrating to Amazon Glue 6.0 by following Migrating from Amazon Glue 5.1 to Amazon Glue 6.0.

Appendix B: JDBC driver upgrades

Appendix C: Connector upgrades

Appendix D: Open table format upgrades

Iceberg upgraded to 1.11.0 with full support for Apache Iceberg version 3 specifications.

VARIANT data type with variant shredding for optimized semi-structured data queries.

Nanosecond-precision timestamps.

Geospatial data types (Geometry and Geography).

Hudi upgraded to 1.1.1.

Continued support for FTA read and write access on Amazon Lake Formation registered tables.

Delta Lake upgraded to 4.2.0.

Continued support for FTA read and write access on Amazon Lake Formation registered tables.

Iceberg native table encryption keys are not supported.

Iceberg multi-argument transforms are not supported.

The new Iceberg v3 data types are only supported with Spark DataFrames. These features will not work with DynamicFrames.

Visual ETL in Amazon Glue Studio does not support the new Iceberg v3 data types. If you want to use these new features with Visual ETL, we recommend migrating your ETL jobs to Amazon SageMaker Unified Studio.

Fine-grained access control (FGAC) is not supported with VARIANT columns.

Iceberg tables created with 'format-version'='3' cannot be read by Athena SQL (error: Cannot read unsupported version 3). Use Iceberg v2 for cross-engine compatibility with Athena.