Table properties - Amazon Kinesis Data Analytics
Apache Flink Time ValuesFlink Connector and Format Properties
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Table properties

In addition to data fields, your Amazon Glue tables provide other information to your Studio notebook using table properties. Kinesis Data Analytics uses the following Amazon Glue table properties:

To add a property to an Amazon Glue table, do the following:

  1. Sign in to the Amazon Web Services Management Console and open the Amazon Glue console at https://console.amazonaws.cn/glue/.

  2. From the list of tables, choose the table that your application uses to store its data connection information. Choose Action, Edit table details.

  3. Under Table Properties, enter kinesisanalytics.proctime for key and user_action_time for Value.

Using Apache Flink time values

Apache Flink provides time values that describe when stream processing events occured, such as Processing Time and Event Time. To include these values in your application output, you define properties on your Amazon Glue table that tell the Kinesis Data Analytics runtime to emit these values into the specified fields.

The keys and values you use in your table properties are as follows:

Timestamp Type Key Value
Processing Time kinesisanalytics.proctime The column name that Amazon Glue will use to expose the value. This column name does not correspond to an existing table column.
Event Time kinesisanalytics.rowtime The column name that Amazon Glue will use to expose the value. This column name corresponds to an existing table column.

kinesisanalytics.watermark.column_name.milliseconds

 The watermark interval in milliseconds

Using Flink Connector and format properties

You provide information about your data sources to your application's Flink connectors using Amazon Glue table properties. Some examples of the properties that Kinesis Data Analytics uses for connectors are as follows:

Connector Type Key Value
Kafka format The format used to deserialize and serialize Kafka messages, e.g. json or csv.
scan.startup.mode The startup mode for the Kafka consumer, e.g. earliest-offset or timestamp.
Kinesis format The format used to deserialize and serialize Kinesis data stream records, e.g. json or csv.
aws.region The Amazon region where the stream is defined.
S3 (Filesystem) format The format used to deserialize and serialize files, e.g. json or csv.
path The Amazon S3 path, e.g. s3://mybucket/.

For more information about other connectors besides Kinesis and Apache Kafka, see your connector's documentation.