Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Table API Connectors

In the Apache Flink programming model, connectors are components that your application uses to read or write data from external sources, such as other Amazon services.

With the Apache Flink Table API, you can use the following types of connectors:

Table API Sources

You create a table source from a data stream. The following code creates a table from an Amazon MSK topic:

//create the table
    final FlinkKafkaConsumer<StockRecord> consumer = new FlinkKafkaConsumer<StockRecord>(kafkaTopic, new KafkaEventDeserializationSchema(), kafkaProperties);
    consumer.setStartFromEarliest();
    //Obtain stream
    DataStream<StockRecord> events = env.addSource(consumer);

    Table table = streamTableEnvironment.fromDataStream(events);

For more information about table sources, see Table & Connectors in the Apache Flink documentation.

Table API Sinks

To write table data to a sink, you create the sink in SQL, and then run the SQL-based sink on the StreamTableEnvironment object.

The following code example demonstrates how to write table data to an Amazon S3 sink:

final String s3Sink = "CREATE TABLE sink_table (" +
    "event_time TIMESTAMP," +
    "ticker STRING," +
    "price DOUBLE," +
    "dt STRING," +
    "hr STRING" +
    ")" +
    " PARTITIONED BY (ticker,dt,hr)" +
    " WITH" +
    "(" +
    " 'connector' = 'filesystem'," +
    " 'path' = '" + s3Path + "'," +
    " 'format' = 'json'" +
    ") ";

    //send to s3
    streamTableEnvironment.executeSql(s3Sink);
    filteredTable.executeInsert("sink_table");

You can use the format parameter to control what format Kinesis Data Analytics uses to write the output to the sink. For information about formats, see Formats in the Apache Flink documentation.

For more information about table sinks, see Table & Connectors in the Apache Flink documentation.

User-Defined Sources and Sinks

You can use existing Apache Kafka connectors for sending data to and from other Amazon services, such as Amazon MSK and Amazon S3. For interacting with other data sources and destinations, you can define your own sources and sinks. For more information, see User-Defined Sources and Sinks in the Apache Flink documentation.