Apache Flink settings - Managed Service for Apache Flink
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.

Apache Flink settings

Managed Service for Apache Flink is an implementation of the Apache Flink framework. Managed Service for Apache Flink uses the default values described in this section. Some of these values can be set by the Managed Service for Apache Flink applications in code, and others cannot be changed.

Managed Service for Apache Flink provides a default Flink configuration consisting of Apache Flink-recommended values for most properties and a few based on common application profiles. For more information about Flink configuration, see Configuration. Service-provided default configuration works for most applications. However, to tweak Flink configuration properties to improve performance for certain applications with high parallelism, high memory and state usage, or enable new debugging features in Apache Flink, you can change certain properties by requesting a support case. For more information, see Amazon Support Center. You can check the current configuration for your application using the Apache Flink Dashboard.

State backend

Managed Service for Apache Flink stores transient data in a state backend. Managed Service for Apache Flink uses the RocksDBStateBackend. Calling setStateBackend to set a different backend has no effect.

We enable the following features on the state backend:

  • Incremental state backend snapshots

  • Asynchronous state backend snapshots

  • Local recovery of checkpoints

For more information about state backends, see State Backends in the Apache Flink Documentation.

Checkpointing

Managed Service for Apache Flink uses a default checkpoint configuration with the following values. Some of these values can be changed using CheckpointConfiguration. You must set CheckpointConfiguration.ConfigurationType to CUSTOM for Managed Service for Apache Flink to use modified checkpointing values.

Setting Can be modified? How Default Value
CheckpointingEnabled Modifiable

Create Application

Update Application

Amazon CloudFormation

True
CheckpointInterval Modifiable

Create Application

Update Application

Amazon CloudFormation

60000
MinPauseBetweenCheckpoints Modifiable

Create Application

Update Application

Amazon CloudFormation

5000
Unaligned checkpoints Modifiable Support case False
Number of Concurrent Checkpoints Not Modifiable N/A 1
Checkpointing Mode Not Modifiable N/A Exactly Once
Checkpoint Retention Policy Not Modifiable N/A On Failure
Checkpoint Timeout Not Modifiable N/A 60 minutes
Max Checkpoints Retained Not Modifiable N/A 1
Checkpoint and Savepoint Location Not Modifiable N/A We store durable checkpoint and savepoint data to a service-owned S3 bucket.

Savepointing

By default, when restoring from a savepoint, the resume operation will try to map all state of the savepoint back to the program you are restoring with. If you dropped an operator, by default, restoring from a savepoint that has data that corresponds to the missing operator will fail. You can allow the operation to succeed by setting the AllowNonRestoredState parameter of the application's FlinkRunConfiguration to true. This will allow the resume operation to skip state that cannot be mapped to the new program.

For more information, see Allowing Non-Restored State in the Apache Flink documentation.

Heap sizes

Managed Service for Apache Flink allocates each KPU 3 GiB of JVM heap, and reserves 1 GiB for native code allocations. For information about increasing your application capacity, see Application scaling in Managed Service for Apache Flink.

For more information about JVM heap sizes, see Configuration in the Apache Flink documentation.

Buffer debloating

Buffer debloating can help applications with high backpressure. If your application experiences failed checkpoints/savepoints, enabling this feature could be useful. To do this, request a support case.

For more information, see The Buffer Debloating Mechanism in the Apache Flink documentation.