Tutorial: Analyze real-time stock data using Amazon Managed Service for Apache Flink
The scenario for this tutorial involves ingesting stock trades into a data stream and writing a simple Amazon Managed Service for Apache Flink application that performs calculations on the stream. You will learn how to send a stream of records to Kinesis Data Streams and implement an application that consumes and processes the records in near real time.
With Amazon Managed Service for Apache Flink, you can use Java or Scala to process and analyze streaming data. The service lets you author and run Java or Scala code against streaming sources to perform time-series analytics, feed real-time dashboards, and create real-time metrics.
You can build Flink applications in Managed Service for Apache Flink using open-source libraries based on Apache Flink
Important
After you create two data streams and an application, your account incurs nominal charges for Kinesis Data Streams and Managed Service for Apache Flink usage because they are not eligible for the Amazon Free Tier. When you are finished with this application, delete your Amazon resources to stop incurring charges.
The code does not access actual stock market data, but instead simulates the stream of stock trades. It does so by using a random stock trade generator. If you have access to a real-time stream of stock trades, you might be interested in deriving useful, timely statistics from that stream. For example, you might want to perform a sliding window analysis where you determine the most popular stock purchased in the last 5 minutes. Or you might want a notification whenever there is a sell order that is too large (that is, it has too many shares). You can extend the code in this series to provide such functionality.
The examples shown use the US West (Oregon) Region, but they work on any of the Amazon Regions that support Managed Service for Apache Flink.
Tasks
Prerequisites for completing the exercises
To complete the steps in this guide, you must have the following:
-
Java Development Kit
(JDK) version 8. Set the JAVA_HOME
environment variable to point to your JDK install location. -
We recommend that you use a development environment (such as Eclipse Java Neon
or IntelliJ Idea ) to develop and compile your application. -
Git Client.
Install the Git client if you haven't already. -
Apache Maven Compiler Plugin
. Maven must be in your working path. To test your Apache Maven installation, enter the following: $ mvn -version
To get started, go to Set up an Amazon account and create an administrator user.