Migrating Amazon Glue for Spark jobs to Amazon Glue version 4.0

It is based on Apache Spark 3.3.0, but includes optimizations in Amazon Glue, and Amazon EMR, such as adaptive query runs, vectorized readers, and optimized shuffles and partition coalescing.

Upgraded JDBC drivers for all Amazon Glue native sources including MySQL, Microsoft SQL Server, Oracle, PostgreSQL, MongoDB, and upgraded Spark libraries and dependencies brought in by Spark 3.3.0.

Updated with a new Amazon Redshift connector and JDBC driver.

Optimized Amazon S3 access with upgraded EMR File System (EMRFS) and enabled Amazon S3-optimized output committers, by default.

Optimized Data Catalog access with partition indexes, pushdown predicates, partition listing, and an upgraded Hive metastore client.

Integration with Lake Formation for governed catalog tables with cell-level filtering and data lake transactions.

Reduced startup latency to improve overall job completion times and interactivity.

Spark jobs are billed in 1-second increments with a 10x lower minimum billing duration—from a 10-minute minimum to a 1-minute minimum.

Native support for open-data lake frameworks with Apache Hudi, Delta Lake, and Apache Iceberg.

Native support for the Amazon S3-based Cloud Shuffle Storage Plugin (an Apache Spark plugin) to use Amazon S3 for shuffling and elastic storage capacity.

Row-level runtime filtering (SPARK-32268).

ANSI enhancements (SPARK-38860).

Error message improvements (SPARK-38781).

Support complex types for Parquet vectorized reader (SPARK-34863).

Hidden file metadata support for Spark SQL (SPARK-37273).

Provide a profiler for Python/Pandas UDFs (SPARK-37443).

Introduce Trigger.AvailableNow for running streaming queries like Trigger.Once in multiple batches (SPARK-36533).

More comprehensive Datasource V2 pushdown capabilities (SPARK-38788).

Migrating from log4j 1 to log4j 2 (SPARK-37814).

Breaking changes

In Amazon Glue Studio, choose Glue 4.0 - Supports Spark 3.3, Scala 2, Python 3 in Glue version.

In the API, choose 4.0 in the GlueVersion parameter in the UpdateJob API operation.

In the console, choose Spark 3.3, Python 3 (Glue Version 4.0) or Spark 3.3, Scala 2 (Glue Version 3.0) in Glue version.

In Amazon Glue Studio, choose Glue 4.0 - Supports Spark 3.3, Scala 2, Python 3 in Glue version.

In the API, choose 4.0 in the GlueVersion parameter in the CreateJob API operation.

Do your job's external Python libraries depend on Python 2.7/3.6?

All existing job parameters and major features that exist in Amazon Glue 3.0 will exist in Amazon Glue 4.0.

Amazon Glue 3.0 uses Amazon EMR-optimized Spark 3.1.1, and Amazon Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

Several Spark changes alone might require revision of your scripts to ensure that removed features are not being referenced.

Amazon Glue 4.0 also features an update to EMRFS and Hadoop. For the specific version, see Appendix A: Notable dependency upgrades.

The Amazon SDK provided in ETL jobs is now upgraded from 1.11 to 1.12.

All Python jobs will be using Python version 3.10. Previously, Python 3.7 was used in Amazon Glue 3.0.

As a result, some pymodules brought out-of-the-box by Amazon Glue are upgraded.

Log4j has been upgraded to Log4j2.

For migrating certain connectors, see Connector and JDBC driver migration for Amazon Glue 4.0.

The Amazon Encryption SDK is upgraded from 1.x to 2.x. Amazon Glue jobs using Amazon Glue security configurations and jobs dependent on the Amazon Encryption SDK dependency provided in runtime are affected. See the instructions for Amazon Glue job migration.

You can safely upgrade an Amazon Glue 2.0/3.0 job to an Amazon Glue 4.0 job because Amazon Glue 2.0/3.0 already contains the Amazon Encryption SDK bridge version.

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

All existing job parameters and major features that exist in Amazon Glue 2.0 will exist in Amazon Glue 4.0.

The EMRFS S3-optimized committer for writing Parquet data into Amazon S3 is enabled by default since Amazon Glue 3.0. However, you can still disable it by setting --enable-s3-parquet-optimized-committer to false.

Amazon Glue 2.0 uses open-source Spark 2.4 and Amazon Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

The Amazon SDK provided in ETL jobs is now upgraded from 1.11 to 1.12.

Amazon Glue 4.0 also features an update to EMRFS, updated JDBC drivers, and inclusions of additional optimizations onto Spark itself provided by Amazon Glue.

Scala is updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is the default version used for Python scripts, as Amazon Glue 2.0 was only using Python 3.7 and 2.7.

Several dependency updates, highlighted in Appendix A: Notable dependency upgrades.

Any extra JAR files supplied in existing Amazon Glue 2.0 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 4.0 from 2.0. You can avoid classpath conflicts in Amazon Glue 4.0 with the --user-jars-first Amazon Glue job parameter.

Amazon Glue 4.0 uses Spark 3.3. Starting with Spark 3.1, there was a change in the behavior of loading/saving of timestamps from/to parquet files. For more details, see Upgrading from Spark SQL 3.0 to 3.1.

We recommend to set the following parameters when reading/writing parquet data that contains timestamp columns. Setting those parameters can resolve the calendar incompatibility issue that occurs during the Spark 2 to Spark 3 upgrade, for both the Amazon Glue Dynamic Frame and Spark Data Frame. Use the CORRECTED option to read the datetime value as it is; and the LEGACY option to rebase the datetime values with regard to the calendar difference during reading.

- Key: --conf
- Value: spark.sql.legacy.parquet.int96RebaseModeInRead=[CORRECTED|LEGACY] --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=[CORRECTED|LEGACY] --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=[CORRECTED|LEGACY]

For migrating certain connectors, see Connector and JDBC driver migration for Amazon Glue 4.0.

The Amazon Encryption SDK is upgraded from 1.x to 2.x. Amazon Glue jobs using Amazon Glue security configurations and jobs dependent on the Amazon Encryption SDK dependency provided in runtime are affected. See these instructions for Amazon Glue job migration:

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

Amazon Glue 1.0 uses open-source Spark 2.4 and Amazon Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

All jobs in Amazon Glue 4.0 will be run with significantly improved startup times. Spark jobs will be billed in 1-second increments with a 10x lower minimum billing duration since startup latency will go from 10 minutes maximum to 1 minute maximum.

Logging behavior has changed significantly in Amazon Glue 4.0, Spark 3.3.0 has a minimum requirement of Log4j2.

Several dependency updates, highlighted in the appendix.

Scala is also updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is also the default version used for Python scripts, as Amazon Glue 0.9 was only using Python 2.

Python 2.7 is not supported with Spark 3.3.0. Any job requesting Python 2 in the job configuration will fail with an IllegalArgumentException.

A new mechanism of installing additional Python modules through pip is available since Amazon Glue 2.0. For more information, see Installing additional Python modules with pip in Amazon Glue 2.0+.

Amazon Glue 4.0 does not run on Apache YARN, so YARN settings do not apply.

Amazon Glue 4.0 does not have a Hadoop Distributed File System (HDFS).

Any extra JAR files supplied in existing Amazon Glue 1.0 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 4.0 from 1.0. We enable Amazon Glue 4.0 with the --user-jars-first Amazon Glue job parameter by default, to avoid this problem.

Amazon Glue 4.0 supports auto scaling. Therefore, the ExecutorAllocationManager metric will be available when auto scaling is enabled.

In Amazon Glue version 4.0 jobs, you specify the number of workers and worker type, but do not specify a maxCapacity.

Amazon Glue 4.0 does not yet support machine learning transforms.

For migrating certain connectors, see Connector and JDBC driver migration for Amazon Glue 4.0.

The Amazon Encryption SDK is upgraded from 1.x to 2.x. Amazon Glue jobs using Amazon Glue security configurations and jobs dependent on the Amazon Encryption SDK dependency provided in runtime are affected. See these instructions for Amazon Glue job migration.

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.0 to 3.1

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

Amazon Glue 0.9 uses open-source Spark 2.2.1 and Amazon Glue 4.0 uses Amazon EMR-optimized Spark 3.3.0.

All jobs in Amazon Glue 4.0 will be run with significantly improved startup times. Spark jobs will be billed in 1-second increments with a 10x lower minimum billing duration because startup latency will go from 10 minutes maximum to 1 minute maximum.

Logging behavior has changed significantly since Amazon Glue 4.0, Spark 3.3.0 has a minimum requirement of Log4j2 as mentioned here (https://spark.apache.org/docs/latest/core-migration-guide.html#upgrading-from-core-32-to-33).

Several dependency updates, highlighted in the appendix.

Scala is also updated to 2.12 from 2.11, and Scala 2.12 is not backward compatible with Scala 2.11.

Python 3.10 is also the default version used for Python scripts, as Amazon Glue 0.9 was only using Python 2.

Amazon Glue 4.0 does not run on Apache YARN, so YARN settings do not apply.

Amazon Glue 4.0 does not have a Hadoop Distributed File System (HDFS).

Any extra JAR files supplied in existing Amazon Glue 0.9 jobs might bring in conflicting dependencies because there were upgrades in several dependencies in 3.0 from 0.9. You can avoid classpath conflicts in Amazon Glue 3.0 with the --user-jars-first Amazon Glue job parameter.

Amazon Glue 4.0 supports auto scaling. Therefore, the ExecutorAllocationManager metric will be available when auto scaling is enabled.

In Amazon Glue version 4.0 jobs, you specify the number of workers and worker type, but do not specify a maxCapacity.

Amazon Glue 4.0 does not yet support machine learning transforms.

For migrating certain connectors, see Connector and JDBC driver migration for Amazon Glue 4.0.

The Amazon Encryption SDK is upgraded from 1.x to 2.x. Amazon Glue jobs using Amazon Glue security configurations and jobs dependent on the Amazon Encryption SDK dependency provided in runtime are affected. See these instructions for Amazon Glue job migration.

Upgrading from Spark SQL 2.2 to 2.3

Upgrading from Spark SQL 2.3 to 2.4

Upgrading from Spark SQL 2.4 to 3.0

Upgrading from Spark SQL 3.0 to 3.1

Upgrading from Spark SQL 3.1 to 3.2

Upgrading from Spark SQL 3.2 to 3.3

Changes in Datetime behavior to be expected since Spark 3.0.

Appendix B: JDBC driver upgrades

Appendix C: Connector upgrades

Spark SQL support improvements:

Any Hudi table created before version 0.10.0 without a primaryKey needs to be recreated with a primaryKey field since version 0.10.0.

Several vulnerabilities (CVEs) were addressed.

Java 8 is natively supported.

If the job is using Arrays of Arrays, with the exception of byte arrays, this scenario can be treated as multidimensional arrays.

The current MongoDB connector supports Spark version 3.1 or later and MongoDB version 4.0 or later.

Due to the connector upgrade, a few property names changed. For example, the URI property name changed to connection.uri. For more information on the current options, see the MongoDB Spark Connector blog.

Using MongoDB 4.0 hosted by Amazon DocumentDB has some functional differences. For more information, see these topics:

The "partitioner" option is restricted to ShardedPartitioner, PaginateIntoPartitionsPartitioner, and SinglePartitionPartitioner. It cannot use default SamplePartitioner and PaginateBySizePartitioner for Amazon DocumentDB because the stage operator does not support the MongoDB API. For more information, see Supported MongoDB APIs, Operations, and Data Types.

Delta Lake now supports time travel in SQL to query older data easily. With this update, time travel is now available both in Spark SQL and through the DataFrame API. Support has been added for the current version of TIMESTAMP in SQL.

Spark 3.3 introduces Trigger.AvailableNow for running streaming queries as an equivalent to Trigger.Once for batch queries. This support is also available when using Delta tables as a streaming source.

Support for SHOW COLUMNS to return the list of columns in a table.

Support for DESCRIBE DETAIL in the Scala and Python DeltaTable API. It retrieves detailed information about a Delta table using either the DeltaTable API or Spark SQL.

Support for returning operation metrics from SQL Delete, Merge, and Update commands. Previously these SQL commands returned an empty DataFrame, now they return a DataFrame with useful metrics about the operation performed.

Optimize performance improvements:

Other notable changes:

Added several performance improvements for scan planning and Spark queries.

Added a common REST catalog client that uses change-based commits to resolve commit conflicts on the service side.

AS OF syntax for SQL time travel queries is supported.

Added merge-on-read support for MERGE and UPDATE queries.

Added support to rewrite partitions using Z-order.

Added a spec and implementation for Puffin, a format for large stats and index blobs, like Theta sketches or bloom filters.

Added new interfaces for consuming data incrementally (both append and changelog scans).

Added support for bulk operations and ranged reads to FileIO interfaces.

Added more metadata tables to show delete files in the metadata tree.

The drop table behavior changed. In Iceberg 0.13.1, running DROP TABLE removes the table from the catalog and deletes the table contents as well. In Iceberg 1.0.0, DROP TABLE only removes the table from the catalog. To delete the table contents use DROP TABLE PURGE.

Parquet vectorized reads are enabled by default in Iceberg 1.0.0. If you want to disable vectorized reads, set read.parquet.vectorization.enabled to false.