Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting IDT for Amazon IoT Greengrass V2

IDT for Amazon IoT Greengrass V2 writes errors to various locations based on the type of errors. IDT writes errors to the console, log files, and test reports.

Where to look for errors

High-level errors are displayed on the console while the test is running, and a summary of the failed tests displays when all tests are complete. awsiotdevicetester_report.xml contains a summary of all the errors that caused a test to fail. IDT stores the log files for each test run in a directory with a UUID for the test execution, displayed on the console during the test run.

The IDT test logs directory is <device-tester-extract-location>/results/<execution-id>/logs/. This directory contains the following files displayed in the table. This is useful for debugging.

Resolving IDT for Amazon IoT Greengrass V2 errors

Before you run IDT for Amazon IoT Greengrass, get the correct configuration files in place. If you receive parsing and configuration errors, your first step is to locate and use a configuration template appropriate for your environment.

If you are still having issues, see the following debugging process.

Alias resolution errors

When you run custom test suites, you might see the following error in the console and in the test_manager.log.

Couldn't resolve placeholders: couldn't do a json lookup: index out of range

This error can occur when the aliases configured in the IDT test orchestrator don't resolve correctly or if the resolved values aren't present in the configuration files. To resolve this error, make sure that your device.json and userdata.json contain the correct information required for your test suite. For information about the configuration required for Amazon IoT Greengrass qualification, see Configure IDT settings to run the Amazon IoT Greengrass qualification suite.

Conflict errors

You might see the following error when you run the Amazon IoT Greengrass qualification suite concurrently on more than one device.

ConflictException: Component [com.example.IDTHelloWorld : 1.0.0] for account [account-id] already exists with state: [DEPLOYABLE] { RespMetadata: { StatusCode: 409, RequestID: “id” }, Message_: “Component [com.example.IDTHelloWorld : 1.0.0] for account [account-id] already exists with state: [DEPLOYABLE]” }

Concurrent test execution isn't yet supported for the Amazon IoT Greengrass qualification suite. Run the qualification suite sequentially for each device.

Could not start test error

You might encounter errors that point to failures that occurred when the test was attempting to start. There are several possible causes, so do the following:

Docker qualification image exists errors

The Docker application manager qualification tests use the amazon/amazon-ec2-metadata-mock container image in Amazon ECR to qualify the device under test.

You might receive the following error if the image is already present in a Docker container on the device under test.

The Docker image amazon/amazon-ec2-metadata-mock:version already exists on the device.

If you previously downloaded this image and ran the amazon/amazon-ec2-metadata-mock container on your device, make sure you remove this image from the device under test before you run the qualification tests.

Failed to read credential

When testing Windows devices, you might encounter the Failed to read credential error in the greengrass.log file if the user that you use to connect to the device under test is not set up in the credential manager on that device.

To resolve this error, configure the user and password for the IDT user in the credential manager on the device under test.

For more information, see Configure user credentials for Windows devices.

Guice errors with PreInstalled Greengrass

While running IDT with PreInstalled Greengrass, if you encounter an error of Guice or ErrorInCustomProvider, check if the file userdata.json has the InstalledDirRootOnDevice set to the Greengrass installation folder. IDT checks for the file effectiveConfig.yaml under <InstallationDirRootOnDevice>/config/effectiveConfig.yaml.

For more information, see Configure user credentials for Windows devices.

Invalid signature exception

When you run Lambda qualification tests, you might encounter the invalidsignatureexception error if your IDT host machine experiences network access issues. Reset your router and run the tests again.

Machine learning qualification errors

When you run machine learning (ML) qualification tests, you might encounter qualification failures if your device doesn't meet the requirements to deploy the Amazon-provided ML components. To troubleshoot ML qualification errors, do the following:

Open Test Framework (OTF) failed deployments

If OTF tests fail to complete the deployment, a likely cause may be the permissions set for the parent folder of TempResourcesDirOnDevice and InstallationDirRootOnDevice. To set this folder's permissions correctly, run the following command. Replace folder-name with the name of the parent folder.

sudo chmod 755 folder-name

Parsing errors

Typos in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quotation mark from your JSON file. IDT performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you still can't locate the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.

Permission denied errors

IDT performs operations on various directories and files in a device under test. Some of these operations require root access. To automate these operations, IDT must be able to run commands with sudo without typing a password.

Follow these steps to allow sudo access without typing a password.

Qualification report generation error

IDT supports the four latest major.minor versions of the Amazon IoT Greengrass V2 qualification suite (GGV2Q) to generate qualification reports that you can submit to Amazon Partner Network to include your devices in the Amazon Partner Device Catalog. Earlier versions of the qualification suite don’t generate qualification reports.

If you have questions about the support policy, contact Amazon Web Services Support.

Required parameter missing error

When IDT adds new features, it might introduce changes to the configuration files. Using an old configuration file might break your configuration. If this happens, the <test_case_id>.log file under /results/<execution-id>/logs explicitly lists all missing parameters. IDT also validates your JSON configuration file schemas to verify that you are using the latest supported version.

Security exception on macOS

When you run IDT on a macOS host computer, it blocks IDT from running. To run IDT, grant a security exception to the executables that is part of IDT runtime functionality. When you see the warning message display on your host computer, do the following for each of the applicable executables:

To grant a security exception to IDT executables

SSH connection errors

When IDT can't connect to a device under test, it logs connection failures in /results/<execution-id>/logs/<test-case-id>.log. SSH messages appear at the top of this log file because connecting to a device under test is one of the first operations that IDT performs.

Most Windows configurations use the PuTTy terminal application to connect to Linux hosts. This application requires that you convert standard PEM private key files into a proprietary Windows format called PPK. If you configure SSH in your device.json file, use PEM files. If you use a PPK file, IDT can't create an SSH connection with the Amazon IoT Greengrass device and can't run tests.

Starting with IDT v4.4.0, if you haven't enabled SFTP on your device under test, then you might see the following error in the log file.

SSH connection failed with EOF

To resolve this error, enable SFTP on your device.

Stream manager qualification errors

When you run stream manager qualification tests, you might see the following error in the com.aws.StreamManagerExport.log file.

Failed to upload data to S3

This error can occur when stream manager uses the Amazon credentials in the ~/root/.aws/credentials file on your device instead of using the environment credentials that IDT exports to the device under test. To prevent this issue, delete the credentials file on your device, and re-run the qualification test.

Timeout errors

You can increase the timeout for each test by specifying a timeout multiplier applied to the default value of each test's timeout. Any value configured for this flag must be greater than or equal to 1.0.

To use the timeout multiplier, use the flag --timeout-multiplier when running the tests. For example:

./devicetester_linux run-suite --suite-id GGV2Q_1.0.0 --pool-id DevicePool1 --timeout-multiplier 2.5

For more information, run run-suite --help.

Some timeout errors occur when IDT test cases can’t be completed because of configuration issues. You can’t resolve these errors by increasing the timeout multiplier. Use the logs from the test run to troubleshoot the underlying configuration issues.

Version check errors

IDT issues the following error when the Amazon user credentials for the IDT user don't have the required IAM permissions.

Failed to check version compatibility

The Amazon user that doesn't have the required IAM permissions.