Amazon Managed Service for Apache Flink 1.19 - Managed Service for Apache Flink
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.

Amazon Managed Service for Apache Flink 1.19

Managed Service for Apache Flink now supports Apache Flink version 1.19.1. This section introduces you to the key new features and changes introduced with Managed Service for Apache Flink support of Apache Flink 1.19.1.

Note

If you are using an earlier supported version of Apache Flink and want to upgrade your existing applications to Apache Flink 1.19.1, you can do so using in-place Apache Flink version upgrades. For more information, see In-place version upgrades for Apache Flink. With in-place version upgrades, you retain application traceability against a single ARN across Apache Flink versions, including snapshots, logs, metrics, tags, Flink configurations, and more.

Apache Flink 1.19.1 introduces improvements in the SQL API, such as named parameters, custom source parallelism, and different state TTLs for various Flink operators.

Supported features and related documentation
Supported features Description Apache Flink documentation reference
SQL API: Support Configuring Different State TTLs using SQL Hint Users can now configure state TTL on stream regular joins and group aggregate. FLIP-373: Configuring Different State TTLs using SQL Hint
SQL API: Support named parameters for functions and call procedures Users can now use named parameters in functions, rather than relying on the order of parameters. FLIP-378: Support named parameters for functions and call procedures
SQL API: Setting parallelism for SQL sources Users can now specify parallelism for SQL sources. FLIP-367: Support Setting Parallelism for Table/SQL Sources
SQL API: Support Session Window TVF Users can now use session window Table-Valued Functions. FLINK-24024: Support session Window TVF
SQL API: Window TVF Aggregation Supports Changelog Inputs Users can now perform window aggregation on changelog inputs. FLINK-20281: Window aggregation supports changelog stream input

Support Python 3.11

Flink now supports Python 3.11, which is 10-60% faster compared to Python 3.10. For more information, see What's New in Python 3.11. FLINK-33030: Add python 3.11 support
Provide metrics for TwoPhaseCommitting sink Users can view statistics around the status of committers in two phase committing sinks. FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

Trace Reporters for job restart and checkpointing

Users can now monitor traces around checkpoint duration and recvery trends. In Amazon Managed Service for Apache Flink, we enable Slf4j trace reporters by default, so users can monitor checkpoint and job traces through application CloudWatch Logs. FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces
Note

You can opt into the following features by submitting a support case:

Opt-in features and related documentation
Opt-in features Description Apache Flink documentation reference
Support using larger checkpointing interval when source is processing backlog This is an opt-in feature, because users must tune the configuration for their specific job requirements. FLIP-309: Support using larger checkpointing interval when source is processing backlog
Redirect System.out and System.err to Java logs This is an opt-in feature. On Amazon Managed Service for Apache Flink, the default behavior is to ignore output from System.out and System.err because best practice in production is to use the native Java logger. FLIP-390: Support System out and err to be redirected to LOG or discarded

For the Apache Flink 1.19.1 release documentation, see Apache Flink Documentation v1.19.1.

Logging Trace Reporter enabled by default

Apache Flink 1.19.1 introduced checkpoint and recovery traces, enabling users to better debug checkpoint and job recovery issues. In Amazon Managed Service for Apache Flink, these traces are logged into the CloudWatch log stream, allowing users to break down the time spent on job initialization, and record historical size of checkpoints.

Default restart strategy is now exponential-delay

In Apache Flink 1.19.1, there are significant improvements to the exponential-delay restart strategy. In Amazon Managed Service for Apache Flink from Flink 1.19.1 onwards, Flink jobs use the exponential-delay restart strategy by default. This means that user jobs will recover quicker from transient errors, but will not overload external systems if job restarts persist.

Specific Flink job configuration using Flink job code is disabled

In Apache Flink 1.18 and earlier versions, all programmatic configuration changes were tolerated by Amazon Managed Service for Apache Flink, with some configurations being silently overridden. From Apache Flink 1.19 and later, we have disabled Flink job configuration changes using Flink job code. If users specify this configuration, their job will throw a ProgramInvocationException listing. For more information, see FlinkRuntimeException: "Not allowed configuration change(s) were detected".

Component Version
Java 11 (recommended)
Python

3.11

Kinesis Data Analytics Flink Runtime (aws-kinesisanalytics-runtime) 1.2.0
Connectors For information about available connectors, see Apache Flink connectors.
Apache Beam (Beam applications only)

There is no compatible Apache Flink Runner for Flink 1.19. For more information, see Flink Version Compatibility.

Apache Beam

There is presently no compatible Apache Flink Runner for Flink 1.19 in Apache Beam. For more information, see Flink Version Compatibility.

Amazon Managed Service for Apache Flink Studio

Studio uses Apache Zeppelin notebooks to provide a single-interface development experience for developing, debugging code, and running Apache Flink stream processing applications. An upgrade is required to Zeppelin’s Flink Interpreter to enable support of Flink 1.19. This work is scheduled with the Zeppelin community and we will update these notes when it is complete. You can continue to use Flink 1.15 with Amazon Managed Service for Apache Flink Studio. For more information, see Creating a Studio notebook.