Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Application Scaling in Kinesis Data Analytics for Apache Flink

You can configure the parallel execution of tasks and the allocation of resources for Amazon Kinesis Data Analytics for Apache Flink to implement scaling. For information about how Apache Flink schedules parallel instances of tasks, see Parallel Execution in the Apache Flink documentation.

Configuring Application Parallelism and ParallelismPerKPU

You configure the parallel execution for your Kinesis Data Analytics application tasks (such as reading from a source or executing an operator) using the following ParallelismConfiguration properties:

For information about setting task parallelism for a specific operator, see Setting the Parallelism: Operator in the Apache Flink documentation.

Allocating Kinesis Processing Units

Kinesis Data Analytics provisions capacity as KPUs. A single KPU provides you with 1 vCPU and 4 GB of memory. For every KPU allocated, 50 GB of running application storage is also provided.

Kinesis Data Analytics calculates the KPUs that are needed to run your application using the Parallelism and ParallelismPerKPU properties, as follows:

Allocated KPUs for the application = Parallelism/ParallelismPerKPU

Kinesis Data Analytics quickly gives your applications resources in response to spikes in throughput or processing activity. It removes resources from your application gradually after the activity spike has passed. To disable the automatic allocation of resources, set the AutoScalingEnabled value to false, as described later in Updating Your Application's Parallelism.

The default limit for KPUs for your application is 32. For instructions on how to request an increase to this limit, see "To request a limit increase" in Service Quotas.

Updating Your Application's Parallelism

This section contains sample requests for API actions that set an application's parallelism. For more examples and instructions for how to use request blocks with API actions, see Kinesis Data Analytics API Example Code.

The following example request for the CreateApplication action sets parallelism when you are creating an application:

{
   "ApplicationName": "string",
   "RuntimeEnvironment":"FLINK-1_15",
   "ServiceExecutionRole":"arn:aws:iam::123456789123:role/myrole",
   "ApplicationConfiguration": { 
      "ApplicationCodeConfiguration":{
      "CodeContent":{
         "S3ContentLocation":{
            "BucketARN":"arn:aws:s3:::mybucket",
            "FileKey":"myflink.jar",
            "ObjectVersion":"AbCdEfGhIjKlMnOpQrStUvWxYz12345"
            }
         },
      "CodeContentType":"ZIPFILE"
   },   
      "FlinkApplicationConfiguration": { 
         "ParallelismConfiguration": { 
            "AutoScalingEnabled": "true",
            "ConfigurationType": "CUSTOM",
            "Parallelism": 4,
            "ParallelismPerKPU": 4
         }
      }
   }
}

The following example request for the UpdateApplication action sets parallelism for an existing application:

{
   "ApplicationName": "MyApplication",
   "CurrentApplicationVersionId": 4,
   "ApplicationConfigurationUpdate": { 
      "FlinkApplicationConfigurationUpdate": { 
         "ParallelismConfigurationUpdate": { 
            "AutoScalingEnabledUpdate": "true",
            "ConfigurationTypeUpdate": "CUSTOM",
            "ParallelismPerKPUUpdate": 4,
            "ParallelismUpdate": 4
         }
      }
   }
}

The following example request for the UpdateApplication action disables parallelism for an existing application:

{
   "ApplicationName": "MyApplication",
   "CurrentApplicationVersionId": 4,
   "ApplicationConfigurationUpdate": { 
      "FlinkApplicationConfigurationUpdate": { 
         "ParallelismConfigurationUpdate": { 
            "AutoScalingEnabledUpdate": "false"
         }
      }
   }
}

Automatic Scaling

Kinesis Data Analytics elastically scales your application’s parallelism to accommodate the data throughput of your source and your operator complexity for most scenarios. Kinesis Data Analytics monitors the resource (CPU) usage of your application, and elastically scales your application's parallelism up or down accordingly:

Kinesis Data Analytics will not reduce your application's CurrentParallelism value to less than your application's Parallelism setting.

When the Kinesis Data Analytics service is scaling your application, it will be in the AUTOSCALING status. You can check your current application status using the DescribeApplication or ListApplications actions. While the service is scaling your application, the only valid API action you can use is StopApplication with the Force parameter set to true.

You can use the AutoScalingEnabled property (part of FlinkApplicationConfiguration ) to enable or disable auto scaling behavior. Your Amazon account is charged for KPUs that Kinesis Data Analytics provisions which is a function of your application's parallelism and parallelismPerKPU settings. An activity spike increases your Kinesis Data Analytics costs.

For information about pricing, see Amazon Kinesis Data Analytics pricing.

Note the following about application scaling:

maxParallelism considerations

For additional information, see Enhanced monitoring and automatic scaling for Apache Flink

For an example, see kda-flink-app-autoscaling.