Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Glue Streaming

Amazon Glue Streaming, a component of Amazon Glue, enables you to efficiently handle streaming data in near real-time, empowering you to carry out crucial tasks such as data ingestion, processing, and machine learning. Using the Apache Spark Streaming framework, Amazon Glue Streaming provides a serverless service that can handle streaming data at scale. Amazon Glue provides various optimizations on top of Apache Spark such as serverless infrastructure, auto-scaling, visual job development, instant-on notebooks for streaming jobs and other performance improvements.

Use cases for streaming

Some common use cases for Amazon Glue Streaming include:

Near-real-time data processing: Amazon Glue Streaming allows organizations to process streaming data in near real-time, enabling them to derive insights and make timely decisions based on the latest information.

Fraud detection: You can utilize Amazon Glue Streaming for real-time analysis of streaming data, making it valuable for detecting fraudulent activities, such as credit card fraud, network intrusion, or online scams. By continuously processing and analyzing incoming data, you can swiftly identify suspicious patterns or anomalies.

Social media analytics: Amazon Glue Streaming can process real-time social media data, such as tweets, posts, or comments, enabling organizations to monitor trends, sentiment analysis, and manage brand reputation in real-time.

Internet of Things (IoT) analytics: Amazon Glue Streaming is suitable for handling and analyzing high-velocity streams of data generated by IoT devices, sensors, and connected machinery. It allows for real-time monitoring, anomaly detection, predictive maintenance, and other IoT analytics use cases.

Clickstream analysis: Amazon Glue Streaming can process and analyze real-time clickstream data from websites or mobile applications. This enables businesses to gain insights into user behavior, personalize user experiences, and optimize marketing campaigns based on real-time clickstream data.

Log monitoring and analysis: Amazon Glue Streaming can continuously process and analyze log data from servers, applications, or network devices in real-time. This helps in detecting anomalies, troubleshooting issues, and monitoring system health and performance.

Recommendation systems: Amazon Glue Streaming can process user activity data in real-time and update recommendation models dynamically. This allows for personalized and real-time recommendations based on user behavior and preferences.

These are some examples of the diverse range of use cases where Amazon Glue Streaming can be applied. Its integration with the Amazon ecosystem and managed services make it a convenient choice for real-time stream processing and analytics in the cloud.

What are the benefits of using Amazon Glue Streaming?

The benefits of using Amazon Glue Streaming are as follows:

When to use Amazon Glue Streaming?

There are many options when it comes to streaming use cases. We recommend Amazon Glue streaming in the following scenarios.

Supported data sources

Amazon Glue Streaming supports the following data sources:

Supported data targets

Amazon Glue Streaming supports a variety of data targets such as:

Enabling real-time mode for streaming jobs

Real-time mode (RTM) is a new execution model for Spark Structured Streaming available in Amazon Glue 6.0. RTM reduces end-to-end latency from seconds or minutes to sub-second. Real-time mode applies only to Spark Structured Streaming jobs. It does not apply to legacy Spark Streaming (DStreams) or other job types.

RTM uses Trigger.RealTime. Tasks run continuously within a batch window (default 5 minutes) and process records as they arrive, rather than accumulating data across intervals. This differs from the default micro-batch model, where forEachBatch/Trigger.ProcessingTime polls, processes, commits, and restarts tasks each interval.

Prerequisites

Before you enable real-time mode, confirm that your job meets the following requirements:

When to use real-time mode

Real-time mode is designed for a specific class of streaming workloads. Consider using real-time mode when:

Continue using micro-batch mode when:

How real-time mode works

The following describes the difference between the micro-batch model and real-time mode:

Both modes use the same checkpoint format and recovery mechanism. The key difference is task lifetime. Micro-batch mode terminates and relaunches tasks every interval. Real-time mode keeps tasks running continuously within a longer batch window.

To enable real-time mode

You enable real-time mode by setting the --enable-real-time-mode job argument to true. You can set this argument in the Amazon Glue console or through the API.

To enable real-time mode (console)

To enable real-time mode (API)

The --enable-real-time-mode flag is stored in the DefaultArguments map of the job definition. You can set it when you create or update a job.

To create a new job (Amazon CLI)

Run the following command:

aws glue create-job \
  --name my-rtm-job \
  --role arn:aws:iam::123456789012:role/MyGlueRole \
  --glue-version 6.0 \
  --worker-type G.1X --number-of-workers 4 \
  --command '{"Name":"gluestreaming","ScriptLocation":"s3://my-bucket/scripts/rtm-job.scala"}' \
  --default-arguments '{
      "--enable-real-time-mode": "true",
      "--job-language": "scala",
      "--class": "GlueApp",
      "--TempDir": "s3://my-bucket/tmp/"
  }' \
  --region us-east-2

To create a new job (boto3)

Use the following code:

import boto3

glue = boto3.client("glue", region_name="us-east-2")

glue.create_job(
    Name="my-rtm-job",
    Role="arn:aws:iam::123456789012:role/MyGlueRole",
    GlueVersion="6.0",
    WorkerType="G.1X",
    NumberOfWorkers=4,
    Command={
        "Name": "gluestreaming",
        "ScriptLocation": "s3://my-bucket/scripts/rtm-job.scala",
    },
    DefaultArguments={
        "--enable-real-time-mode": "true",
        "--job-language": "scala",
        "--class": "GlueApp",
        "--TempDir": "s3://my-bucket/tmp/",
    },
)

To update an existing job (Amazon CLI)

Run the following command:

aws glue update-job \
  --job-name my-existing-job \
  --job-update '{
      "GlueVersion": "6.0",
      "DefaultArguments": {
          "--enable-real-time-mode": "true",
          "--job-language": "scala"
      }
  }'

Writing your streaming script

The job argument declares the intent to use real-time mode. Your script selects the trigger.

The following Scala example shows a streaming query that uses Trigger.RealTime:

import org.apache.spark.sql.streaming.Trigger

val query = df.writeStream
  .format("kafka")
  .outputMode("update")
  .trigger(Trigger.RealTime(60000L))  // checkpoint interval in milliseconds
  .start()

query.awaitTermination()

Trigger.RealTime takes a checkpoint interval in milliseconds. Update output mode is required. Append mode throws OUTPUT_MODE_NOT_SUPPORTED.

You can mix modes in one script as long as the flag is set:

dfA.writeStream.outputMode("update").trigger(Trigger.RealTime(60000L)).start()
dfB.writeStream.outputMode("append").trigger(Trigger.ProcessingTime("30 seconds")).start()

Behavior when the flag is missing

The following describes how the job behaves when the --enable-real-time-mode flag is not set:

Considerations and limitations

Consider the following when you use real-time mode: