Working with Spark jobs in Amazon Glue

Provides information on Amazon Glue for Spark ETL jobs.

Spark Declarative Pipelines

Spark Declarative Pipelines (SDP) is a declarative framework for building batch and streaming data pipelines in Amazon Glue 6.0. With SDP, you define what your data should look like using SQL or Python, and the framework automatically determines the execution plan, resolves dependencies between datasets, and runs independent branches in parallel.

SDP simplifies pipeline development by eliminating imperative boilerplate code for reading, writing, catalog registration, and execution ordering. You focus on business transformations while the framework handles the pipeline infrastructure.

SDP is available in Amazon Glue version 6.0 and later.

SDP concepts

A pipeline consists of a YAML manifest file (spark-pipeline.yml) and one or more SQL or Python transformation files. SDP automatically:

Dataset types

Three dataset types are available in SDP:

Prerequisites

To use SDP, you need the following:

aws glue create-database --database-input '{
  "Name":"my_pipeline_db",
  "LocationUri":"s3://my-bucket/warehouse/my_pipeline_db"
}'

Creating a pipeline

To create an SDP pipeline, complete the following steps.

Step 1: Create the pipeline YAML

Create a file named spark-pipeline.yml:

name: my_analytics_pipeline
catalog: spark_catalog
database: analytics_db
storage: s3://my-bucket/pipeline-storage/
libraries:
  - glob:
      include: transformations/**
configuration:
  spark.sql.shuffle.partitions: "4"

The following table describes the pipeline YAML fields.

Step 2: Write transformations

Create transformation files in a transformations/ directory. You can use SQL, Python, or both in the same pipeline.

SQL example (transformations/silver.sql):

CREATE MATERIALIZED VIEW silver_sales AS
SELECT *, UPPER(region) as clean_region
FROM bronze_sales
WHERE amount > 0;

CREATE MATERIALIZED VIEW gold_summary AS
SELECT clean_region, COUNT(*) as order_count, SUM(amount) as total_revenue
FROM silver_sales
GROUP BY clean_region;

Python example (transformations/bronze.py):

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()

@dp.materialized_view(comment="Raw sales data from S3")
def bronze_sales() -> DataFrame:
    return spark.read.format("csv").option("header", "true") \
        .option("inferSchema", "true") \
        .load("s3://source-bucket/raw-data/sales/")

Python streaming table example (transformations/events.py):

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()

dp.create_streaming_table(
    "streaming_events",
    comment="Incremental event ingestion",
    schema="event_id STRING, event_type STRING, timestamp LONG, payload STRING"
)

@dp.append_flow(target="streaming_events")
def ingest_events() -> DataFrame:
    return (
        spark.readStream.format("json")
        .schema("event_id STRING, event_type STRING, timestamp LONG, payload STRING")
        .load("s3://source-bucket/events/")
    )

Step 3: Upload to Amazon S3

Upload your pipeline files to Amazon S3 as either:

Step 4: Create and run the Amazon Glue job

Create a Amazon Glue job with the following parameters:

The following example creates an SDP job using the Amazon CLI:

aws glue create-job \
  --name my-sdp-pipeline \
  --role arn:aws:iam::123456789012:role/MyGlueRole \
  --glue-version 6.0 \
  --worker-type G.1X --number-of-workers 2 \
  --command '{"Name":"glueetl","ScriptLocation":"s3://my-bucket/pipelines/my_pipeline.zip"}' \
  --default-arguments '{
      "--enable-spark-declarative-pipeline": "true",
      "--enable-glue-datacatalog": "true"
  }'

Running pipelines

You run SDP pipelines using StartJobRun. You can control execution behavior with job arguments passed at run time.

Run modes

Pass the following arguments to StartJobRun to control pipeline execution:

Using Iceberg tables with SDP

For streaming tables that require cross-run incremental processing, use Apache Iceberg. Hive-managed streaming tables store metadata locally and do not persist across job runs.

To configure Iceberg, add the following to your spark-pipeline.yml configuration section:

configuration:
  spark.sql.catalog.spark_catalog: "org.apache.iceberg.spark.SparkSessionCatalog"
  spark.sql.catalog.spark_catalog.type: "hadoop"
  spark.sql.catalog.spark_catalog.warehouse: "s3://my-bucket/iceberg-warehouse"

With Iceberg configured, you get the following benefits:

Considerations and limitations

Consider the following when you use SDP:

Migrating from imperative scripts to SDP

You can migrate existing imperative Spark scripts to SDP incrementally:

SDP can reference any table accessible through the SparkSession, including existing Data Catalog tables, external tables, and cross-database references.