Development Troubleshooting - Amazon Kinesis Data Analytics
Apache Flink Flame GraphsCredential Provider Issue with EFO connector 1.15.2 Applications with unsupported Kinesis connectors Compile Error: "Could not resolve dependencies for project"Invalid Choice: "kinesisanalyticsv2"UpdateApplication Action Isn't Reloading Application Code
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Development Troubleshooting

Apache Flink Flame Graphs

Flame Graphs are enabled by default on applications in Kinesis Data Analytics for Apache Flink versions that support it. Flame Graphs may affect application performance if you keep the graph open, as mentioned in Flink documentation.

If you want to disable Flame Graphs for your application, create a case to request it to be disabled for your application ARN. For more information, see the Amazon Support Center.

Credential Provider Issue with EFO connector 1.15.2

There is a known issue with Kinesis Data Streams EFO connector versions up to 1.15.2 where the FlinkKinesisConsumer is not respecting Credential Provider configuration. Valid configurations are being disregarded due to the issue, which results in the AUTO credential provider being used. This can cause a problem using cross-account access to Kinesis using EFO connector.

To resolve this error please use EFO connector version 1.15.3 or higher.

Applications with unsupported Kinesis connectors

Kinesis Data Analytics for Apache Flink version 1.15 will automatically reject applications from starting or updating if they are using unsupported Kinesis Connector versions (pre-version 1.15.2) bundled into application JARs or archives (ZIP).

Rejection Error

You will see the following error when submitting create / update application calls through:

An error occurred (InvalidArgumentException) when calling the CreateApplication operation: An unsupported Kinesis connector version has been detected in the application. Please update flink-connector-kinesis to any version equal to or newer than 1.15.2.
For more information refer to connector fix: https://issues.apache.org/jira/browse/FLINK-23528

Steps to remediate

  • Update the application’s dependency on flink-connector-kinesis. If you are using Maven as your project’s build tool, follow Update a Maven dependency . If you are using Gradle, follow Update a Gradle dependency .

  • Repackage the application.

  • Upload to an Amazon S3 bucket.

  • Resubmit the create / update application request with the revised application just uploaded to the Amazon S3 bucket.

  • If you continue to see the same error message, re-check your application dependencies. If the problem persists please create a support ticket.

Update a Maven dependency

  1. Open the project’s pom.xml.

  2. Find the project’s dependencies. They look like:

    <project>

    ...

    <dependencies>

        ...

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kinesis</artifactId>
        </dependency>

        ...

    </dependencies>

    ...

</project>

  3. Update flink-connector-kinesis to a version that is equal to or newer than 1.15.2. For instance:

    <project>

    ...

    <dependencies>

        ...

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kinesis</artifactId>
            <version>1.15.2</version>
        </dependency>

        ...

    </dependencies>

    ...

</project>

Update a Gradle dependency

  1. Open the project’s build.gradle (or build.gradle.kts for Kotlin applications).

  2. Find the project’s dependencies. They look like:

    ...

dependencies {

    ...

    implementation("org.apache.flink:flink-connector-kinesis")

    ...

}

...

  3. Update flink-connector-kinesis to a version that is equal to or newer than 1.15.2. For instance:

    ...

dependencies {

    ...

    implementation("org.apache.flink:flink-connector-kinesis:1.15.2")

    ...

}

...

Compile Error: "Could not resolve dependencies for project"

In order to compile the Kinesis Data Analytics for Apache Flink sample applications, you must first download and compile the Apache Flink Kinesis connector and add it to your local Maven repository. If the connector hasn't been added to your repository, a compile error similar to the following appears:

Could not resolve dependencies for project your project name: Failure to find org.apache.flink:flink-connector-kinesis_2.11:jar:1.8.2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

To resolve this error, you must download the Apache Flink source code (version 1.8.2 from https://flink.apache.org/downloads.html) for the connector. For instructions about how to download, compile, and install the Apache Flink source code, see Using the Apache Flink Kinesis Streams Connector with previous Apache Flink versions.

Invalid Choice: "kinesisanalyticsv2"

To use v2 of the Kinesis Data Analytics API, you need the latest version of the Amazon Command Line Interface (Amazon CLI).

For information about upgrading the Amazon CLI, see Installing the Amazon Command Line Interface in the Amazon Command Line Interface User Guide.

UpdateApplication Action Isn't Reloading Application Code

The UpdateApplication action will not reload application code with the same file name if no S3 object version is specified. To reload application code with the same file name, enable versioning on your S3 bucket, and specify the new object version using the ObjectVersionUpdate parameter. For more information about enabling object versioning in an S3 bucket, see Enabling or Disabling Versioning.