Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.
Amazon Managed Service for Apache Flink 1.18
Managed Service for Apache Flink now supports Apache Flink version 1.18.1. Learn about the key new features and changes introduced with Managed Service for Apache Flink support of Apache Flink 1.18.1.
Note
If you are using an earlier supported version of Apache Flink and want to upgrade
your existing applications to Apache Flink 1.18.1, you can do so using in-place
Apache Flink version upgrades. With in-place version upgrades, you retain
application traceability against a single ARN across Apache Flink versions,
including snapshots, logs, metrics, tags, Flink configurations, and more. You can
use this feature in RUNNING and READY state. For more
information, see Use in-place version upgrades for Apache
Flink.
Supported features with Apache Flink documentation references | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Supported Features | Description | Apache Flink documentation reference | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Opensearch connector | This connector includes a sink that provides at-least-once guarantees. | github: Opensearch Connector |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Amazon DynamoDB connector | This connector includes a sink that provides at-least-once guarantees. | Amazon DynamoDB Sink |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| MongoDB connector | This connector includes a source and sink that provide at-least-once guarantees. | MongoDB Connector |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Decouple Hive with Flink planner |
You can use the Hive dialect directly without the extra JAR swapping. |
FLINK-26603: Decouple Hive with Flink planner |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Disable WAL in RocksDBWriteBatchWrapper by default |
This provides faster recovery times. |
FLINK-32326: Disable WAL in RocksDBWriteBatchWrapper by default |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Improve the watermark aggregation performance when enabling the watermark alignment |
Improves the watermark aggregation performance when enabling the watermark alignment, and adds the related benchmark. |
FLINK-32524: Watermark aggregation performance |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Make watermark alignment ready for production use | Removes risk of large jobs overloading JobManager |
FLINK-32548: Make watermark alignment ready |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Configurable RateLimitingStratey for Async Sink |
RateLimitingStrategy lets you configure the decision of what to scale, when to scale, and how much to scale. |
FLIP-242: Introduce configurable RateLimitingStrategy for Async
Sink |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Bulk fetch table and column statistics |
Improved query performance. |
FLIP-247: Bulk fetch of table and column statistics for given partitions |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
For the Apache Flink 1.18.1 release documentation, see
Apache Flink 1.18.1 Release Announcement
Changes in Amazon Managed Service for Apache Flink with Apache Flink 1.18
Akka replaced with Pekko
Apache Flink replaced Akka with Pekko in Apache Flink 1.18. This change is fully
supported in Managed Service for Apache Flink from Apache Flink 1.18.1 and later. You don't need to
modify your applications as a result of this change. For more information, see
FLINK-32468:
Replace Akka by Pekko
Support PyFlink Runtime execution in Thread Mode
This Apache Flink change introduces a new execution mode for the Pyflink Runtime framework, Process Mode. Process Mode can now execute Python user-defined functions in the same thread instead of a separate process.
Backported bug fixes
Amazon Managed Service for Apache Flink backports fixes from the Flink community for critical issues. This means that the runtime differs from the Apache Flink 1.18.1 release. Following is a list of bug fixes that we have backported:
Backported bug fixes | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Apache Flink JIRA link | Description | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-33863 |
This fix addresses the issue when a state restore fails for compressed snapshots. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-34063 |
This fix addresses the issue when source operators lose splits when snapshot compression is enabled. Apache Flink offers optional compression (default: off) for all checkpoints and savepoints. Apache Flink identified a bug in Flink 1.18.1 where the operator state couldn't be properly restored when snapshot compression was enabled. This could result in either data loss or inability to restore from checkpoint. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-35069 |
This fix addresses the issue when a Flink job gets stuck triggering a timer at the end of a window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-35097 |
This fix addresses the pissue of duplicate records in a Table API Filesystem connector with the raw format. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-34379 |
This fix addresses the issue of an OutOfMemoryError when enabling dynamic table filtering. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-28693 |
This fix addresses the issue of the Table API being unable to generate a graph if the watermark has a columnBy expression. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| FLINK-35217 |
This fix addresses the issue of a corrupted checkpoint during a specific Flink job failure mode. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Components
| Component | Version |
|---|---|
| Java | 11 (recommended) |
| Scala |
Since version 1.15, Flink is Scala-agnostic. For reference, MSF Flink 1.18 has been verified against Scala 3.3 (LTS). |
| Managed Service for Apache Flink Flink Runtime (aws-kinesisanalytics-runtime) | 1.2.0 |
| Amazon Kinesis Connector (flink-connector-kinesis)[Source] |
4.2.0-1.18 |
| Amazon Kinesis Connector (flink-connector-kinesis)[Sink] |
4.2.0-1.18 |
| Apache Beam (Beam applications only) |
Earlier and up to version 2.75.0. For more information, see
Flink Version Compatibility |
Known issues
Amazon Managed Service for Apache Flink Studio
Studio uses Apache Zeppelin notebooks to provide a single-interface development experience for developing, debugging code, and running Apache Flink stream processing applications. An upgrade is required to Zeppelin’s Flink Interpreter to enable support of Flink 1.18. This work is scheduled with the Zeppelin community and we will update these notes when it is complete. You can continue to use Flink 1.15 with Amazon Managed Service for Apache Flink Studio. For more information, see Creating a Studio notebook.
Incorrect watermark idleness when subtask is backpressured
There is a known issue in watermark generation when a subtask is backpressured,
which has been fixed from Flink 1.19 and later. This can show up as a spike in the
number of late records when a Flink job graph is backpressured. We recommend that
you upgrade to the latest Flink version to pull in this fix. For more information,
see Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked