Amazon Managed Service for Apache Flink 1.18 - Managed Service for Apache Flink
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.

Amazon Managed Service for Apache Flink 1.18

Managed Service for Apache Flink now supports Apache Flink version 1.18.1. Learn about the key new features and changes introduced with Managed Service for Apache Flink support of Apache Flink 1.18.1.

Note

If you are using an earlier supported version of Apache Flink and want to upgrade your existing applications to Apache Flink 1.18.1, you can do so using in-place Apache Flink version upgrades. With in-place version upgrades, you retain application traceability against a single ARN across Apache Flink versions, including snapshots, logs, metrics, tags, Flink configurations, and more. You can use this feature in RUNNING and READY state. For more information, see In-place version upgrades for Apache Flink.

Supported Features Description Apache Flink documentation reference
Opensearch connector This connector includes a sink that provides at-least-once guarantees. github: Opensearch Connector
Amazon DynamoDB connector This connector includes a sink that provides at-least-once guarantees. Amazon DynamoDB Sink
MongoDB connector This connector includes a source and sink that provide at-least-once guarantees. MongoDB Connector
Decouple Hive with Flink planner

You can use the Hive dialect directly without the extra JAR swapping.

FLINK-26603: Decouple Hive with Flink planner
Disable WAL in RocksDBWriteBatchWrapper by default

This provides faster recovery times.

FLINK-32326: Disable WAL in RocksDBWriteBatchWrapper by default

Improve the watermark aggregation performance when enabling the watermark alignment

Improves the watermark aggregation performance when enabling the watermark alignment, and adds the related benchmark.

FLINK-32524: Watermark aggregation performance
Make watermark alignment ready for production use Removes risk of large jobs overloading JobManager FLINK-32548: Make watermark alignment ready

Configurable RateLimitingStratey for Async Sink

RateLimitingStrategy lets you configure the decision of what to scale, when to scale, and how much to scale.

FLIP-242: Introduce configurable RateLimitingStrategy for Async Sink

Bulk fetch table and column statistics

Improved query performance. FLIP-247: Bulk fetch of table and column statistics for given partitions

For the Apache Flink 1.18.1 release documentation, see Apache Flink 1.18.1 Release Announcement.

Akka replaced with Pekko

Apache Flink replaced Akka with Pekko in Apache Flink 1.18. This change is fully supported in Managed Service for Apache Flink from Apache Flink 1.18.1 and later. You don't need to modify your applications as a result of this change. For more information, see FLINK-32468: Replace Akka by Pekko.

Support PyFlink Runtime execution in Thread Mode

This Apache Flink change introduces a new execution mode for the Pyflink Runtime framework, Process Mode. Process Mode can now execute Python user-defined functions in the same thread instead of a separate process.

Component Version
Java 11 (recommended)
Scala

Since version 1.15, Flink is Scala-agnostic. For reference, MSF Flink 1.18 has been verified against Scala 3.3 (LTS).

Managed Service for Apache Flink Flink Runtime (aws-kinesisanalytics-runtime) 1.2.0
Amazon Kinesis Connector (flink-connector-kinesis)[Source] 4.2.0-1.18
Amazon Kinesis Connector (flink-connector-kinesis)[Sink] 4.2.0-1.18
Apache Beam (Beam applications only)

Earlier and up to version 2.75.0. For more information, see Flink Version Compatibility.

State compression in Apache Flink 1.18.1

Apache Flink offers optional compression (default: off) for all checkpoints and savepoints. Apache Flink identified a bug in Flink 1.18.1 where the operator state couldn't be properly restored when snapshot compression was enabled. This could result in either data loss or inability to restore from checkpoint. For more information, see FLINK-34063: When snapshot compression is enabled, rescaling of a source operator leads to some splits getting lost.

To resolve this, Amazon Managed Service for Apache Flink has backported the fix that will be included in future versions of Apache Flink. For more information, see github: Always flush compression buffers.

Amazon Managed Service for Apache Flink Studio

Studio uses Apache Zeppelin notebooks to provide a single-interface development experience for developing, debugging code, and running Apache Flink stream processing applications. An upgrade is required to Zeppelin’s Flink Interpreter to enable support of Flink 1.18. This work is scheduled with the Zeppelin community and we will update these notes when it is complete. You can continue to use Flink 1.15 with Amazon Managed Service for Apache Flink Studio. For more information, see Creating a Studio notebook.