For new projects, we recommend that you use the new Managed Service for Apache Flink Studio over Kinesis Data Analytics for SQL Applications. Managed Service for Apache Flink Studio combines ease of use with advanced analytical capabilities, enabling you to build sophisticated stream processing applications in minutes.
Monitoring for SQL Applications
Monitoring is an important part of maintaining the reliability, availability, and performance of and your application. You should collect monitoring data from all of the parts of your Amazon solution so that you can more easily debug a multipoint failure if one occurs. Before you start monitoring , however, you should create a monitoring plan that includes answers to the following questions:
-
What are your monitoring goals?
-
What resources will you monitor?
-
How often will you monitor these resources?
-
What monitoring tools will you use?
-
Who will perform the monitoring tasks?
-
Who should be notified when something goes wrong?
The next step is to establish a baseline for normal performance in your environment, by measuring performance at various times and under different load conditions. As you monitor , you can store historical monitoring data. If you do, you can compare it with current performance data, identify normal performance patterns and performance anomalies, and devise methods to address issues.
With , you monitor the application. The application processes data streams (input or output), both of which include identifiers which you can use to narrow your search on CloudWatch logs. For information about how processes data streams, see Amazon Kinesis Data Analytics for SQL Applications: How It Works.
The most important metric is the millisBehindLatest, which indicates how far
behind an application is reading from the streaming source. In a typical case, the
milliseconds behind should be at or near zero. It is common for brief spikes to appear,
which appears as an increase in millisBehindLatest.
We recommend that you set up a CloudWatch alarm that triggers when the application is behind by more than an hour reading the streaming source. For some use cases that require very close to real-time processing, such as emitting processed data to a live application, you might choose to set the alarm at a lower value, such as five minutes.