Amazon Managed Service for Apache Flink was previously known as Amazon Kinesis Data Analytics for Apache Flink.
Using connectors to move data in Managed Service for Apache Flink with the DataStream API
In the Amazon Managed Service for Apache Flink DataStream API, connectors are software components that move data into and out of a Managed Service for Apache Flink application. Connectors are flexible integrations that enable you to read from files and directories. Connectors consist of complete modules for interacting with Amazon services and third-party systems.
Types of connectors include the following:
Sources: Provide data to your application from a Kinesis data stream, file, or other data source.
Sinks: Send data from your application to a Kinesis data stream, Firehose stream, or other data destination.
Asynchronous I/O: Provides asynchronous access to a data source (such as a database) to enrich stream events.
Available connectors
The Apache Flink framework contains connectors for accessing data from a variety of sources. For information about connectors available in the Apache Flink framework, see Connectors
Warning
If you have applications running on Flink 1.6, 1.8, 1.11 or 1.13 and would like to run in Middle East (UAE), Asia Pacific (Hyderabad), Israel (Tel Aviv), Europe (Zurich), Middle East (UAE), Asia Pacific (Melbourne) or Asia Pacific (Jakarta) Regions you may need to rebuild your application archive with an updated connector or upgrade to Flink 1.18.
Apache Flink connectors are stored in their own open source repositories. If
you're upgrading to version 1.18 or later, you must to update your dependencies. To
access the repository for Apache Flink Amazon connectors, see flink-connector-aws
Following are recommended guidelines:
Connector upgrades | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Flink version | Connector used | Resolution | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.15 | EFO |
When upgrading to Amazon Managed Service for Apache Flink version 1.15, make sure that you are using the most recent EFO connector. That must be any version 1.15.3 or later. For more information, see: |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.15 | Amazon Data Firehose Sink |
When upgrading to Amazon Managed Service for Apache Flink version 1.15, make sure that you are using the most recent Amazon Data Firehose Sink. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.15 | Kafka connectors |
When upgrading to Amazon Managed Service for Apache Flink version 1.15, make sure that you
are using the most recent Kafka connector APIs. Apache Flink has
deprecated FlinkKafkaConsumer |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.6 - 1.13 | Firehose |
Your application depends on an outdated version of Firehose connector that is not aware of newer Amazon Regions. Rebuild your application archive with Firehose connector version 2.1.0. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.8 | Kinesis |
Your application depends on an outdated version of Flink Kinesis connector that is not aware of newer Amazon Regions. Rebuild your application archive with Flink Kinesis connector version 1.6.1. https://github.com/awslabs/amazon-kinesis-connector-flink/tree/1.6.1 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.11 | Kinesis |
Your application depends on an outdated version of Flink Kinesis connector that is not aware of newer Amazon Regions. Rebuild your application archive with Flink Kinesis connector version 2.4.1. https://github.com/awslabs/amazon-kinesis-connector-flink/tree/2.4.1 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 1.6 and 1.13 | Kinesis |
Your application depends on an outdated version of Flink Kinesis connector that is not aware of newer Amazon Regions. Unfortunately, Flink no longer releases patches or bug fixes for 1.6/1.13 connectors. We suggest updating to Flink 1.15 by rebuilding your application archive with Flink 1.15. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||