Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Developing and testing Amazon Glue job scripts locally

When you develop and test your Amazon Glue for Spark job scripts, there are multiple available options:

You can choose any of the above options based on your requirements.

If you prefer no code or less code experience, the Amazon Glue Studio visual editor is a good choice.

If you prefer an interactive notebook experience, Amazon Glue Studio notebook is a good choice. For more information, see Using Notebooks with Amazon Glue Studio and Amazon Glue. If you want to use your own local environment, interactive sessions is a good choice. For more information, see Using interactive sessions with Amazon Glue.

If you prefer local/remote development experience, the Docker image is a good choice. This helps you to develop and test Amazon Glue for Spark job scripts anywhere you prefer without incurring Amazon Glue cost.

If you prefer local development without Docker, installing the Amazon Glue ETL library directory locally is a good choice.

Developing using Amazon Glue Studio

The Amazon Glue Studio visual editor is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in Amazon Glue. You can visually compose data transformation workflows and seamlessly run them on Amazon Glue's Apache Spark-based serverless ETL engine. You can inspect the schema and data results in each step of the job. For more information, see the Amazon Glue Studio User Guide.

Developing using interactive sessions

Interactive sessions allow you to build and test applications from the environment of your choice. For more information, see Using interactive sessions with Amazon Glue.

Develop and test Amazon Glue jobs locally using a Docker image

For a production-ready data platform, the development process and CI/CD pipeline for Amazon Glue jobs is a key topic. You can flexibly develop and test Amazon Glue jobs in a Docker container. Amazon Glue hosts Docker images on Docker Hub to set up your development environment with additional utilities. You can use your preferred IDE, notebook, or REPL using Amazon Glue ETL library. This topic describes how to develop and test Amazon Glue version 5.0 jobs in a Docker container using a Docker image.

Available Docker images

The following Docker images are available for Amazon Glue on Amazon ECR:.

In this example, we use public.ecr.aws/glue/aws-glue-libs:5 and run the container on a local machine (Mac, Windows, or Linux). This container image has been tested for Amazon Glue version 5.0 Spark jobs. The image contains the following:

To set up your container, pull the image from ECR Public Gallery and then run the container. This topic demonstrates how to run your container with the following methods, depending on your requirements:

Prerequisites

Before you start, make sure that Docker is installed and the Docker daemon is running. For installation instructions, see the Docker documentation for Mac or Linux. The machine running the Docker hosts the Amazon Glue container. Also make sure that you have at least 7 GB of disk space for the image on the host running the Docker.

For more information about restrictions when developing Amazon Glue code locally, see Local development restrictions .

Configuring Amazon

To enable Amazon API calls from the container, set up Amazon credentials by following steps. In the following sections, we will use this Amazon named profile.

In the following sections, we use this Amazon named profile.

If you’re running Docker on Windows, choose the Docker icon (right-click) and choose Switch to Linux containers before pulling the image.

Run the following command to pull the image from ECR Public:

docker pull public.ecr.aws/glue/aws-glue-libs:5 

Run the container

You can now run a container using this image. You can choose any of following based on your requirements.

spark-submit

You can run an Amazon Glue job script by running the spark-submit command on the container.

REPL shell (Pyspark)

You can run REPL (read-eval-print loops) shell for interactive development. Run the following command to execute the PySpark command on the container to start the REPL shell:

$ docker run -it --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_pyspark \
    public.ecr.aws/glue/aws-glue-libs:5 \
    pyspark

You will see the following output:

Python 3.11.6 (main, Jan  9 2025, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.4-amzn-0
      /_/

Using Python version 3.11.6 (main, Jan  9 2025 00:00:00)
Spark context Web UI available at None
Spark context available as 'sc' (master = local[*], app id = local-1740643079929).
SparkSession available as 'spark'.
>>> 

With this REPL shell, you can code and test interactively.

Pytest

For unit testing, you can use pytest for Amazon Glue Spark job scripts. Run the following commands for preparation.

$ WORKSPACE_LOCATION=/local_path_to_workspace
$ SCRIPT_FILE_NAME=sample.py
$ UNIT_TEST_FILE_NAME=test_sample.py
$ mkdir -p ${WORKSPACE_LOCATION}/tests
$ vim ${WORKSPACE_LOCATION}/tests/${UNIT_TEST_FILE_NAME}

Run the following command to run pytest using docker run:

$ docker run -i --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \
    --workdir /home/hadoop/workspace \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_pytest \
    public.ecr.aws/glue/aws-glue-libs:5 \
    -c "python3 -m pytest --disable-warnings"

Once pytest finishes executing unit tests, your output will look something like this:

============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/hadoop/workspace
plugins: integration-mark-0.2.0
collected 1 item

tests/test_sample.py .                                                   [100%]

======================== 1 passed, 1 warning in 34.28s =========================

Setting up the container to use Visual Studio Code

To set up the container with Visual Studio Code, complete the following steps:

To set up the container:

Changes between Amazon Glue 4.0 and Amazon Glue 5.0 Docker image

The major changes between the Amazon Glue 4.0 and Amazon Glue 5.0 Docker image:

The above list is specific to the Docker image. To learn more about Amazon Glue 5.0 updates, see Introducing Amazon Glue 5.0 for Apache Spark and Migrating Amazon Glue for Spark jobs to Amazon Glue version 5.0.

Considerations

Keep in mind that the following features are not supported when using the Amazon Glue container image to develop job scripts locally.

Appendix: Adding JDBC drivers and Java libraries

To add JDBC driver not currently available in the container, you can create a new directory under your workspace with JAR files you need and mount the directory to /opt/spark/jars/ in docker run command. JAR files found under /opt/spark/jars/ within the container are automatically added to Spark Classpath and will be available for use during job run.

For example, use the following docker run command to add JDBC driver jars to PySpark REPL shell.

docker run -it --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \
    -v $WORKSPACE_LOCATION/jars/:/opt/spark/jars/ \
    --workdir /home/hadoop/workspace \
    -e AWS_PROFILE=$PROFILE_NAME \
    --name glue5_jdbc \
    public.ecr.aws/glue/aws-glue-libs:5 \
    pyspark

As highlighted in Considerations, customJdbcDriverS3Path connection option cannot be used to import a custom JDBC driver from Amazon S3 in Amazon Glue container images.