Use IDT to run the Amazon IoT Greengrass qualification suite - Amazon IoT Greengrass
Test suite versionsTest group descriptions
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Use IDT to run the Amazon IoT Greengrass qualification suite

You can use Amazon IoT Device Tester (IDT) for Amazon IoT Greengrass to verify that the Amazon IoT Greengrass Core software runs on your hardware and can communicate with the Amazon Web Services Cloud. It also performs end-to-end tests with Amazon IoT Core. For example, it verifies that your device can send and receive MQTT messages and process them correctly.

Because Amazon IoT Greengrass Version 1 has been moved into maintenance mode, IDT for Amazon IoT Greengrass V1 no longer generates signed qualification reports. If you want to add your hardware to the Amazon Partner Device Catalog, run the Amazon IoT Greengrass V2 qualification suite to generate test reports that you can submit to Amazon IoT. For more information, see Amazon Device Qualification Program and Supported versions of IDT for Amazon IoT Greengrass V2.

In addition to testing devices, IDT for Amazon IoT Greengrass creates resources (for example, Amazon IoT things, Amazon IoT Greengrass groups, Lambda functions, and so on) in your Amazon Web Services account to facilitate the qualification process.

To create these resources, IDT for Amazon IoT Greengrass uses the Amazon credentials configured in the config.json file to make API calls on your behalf. These resources are provisioned at various times during a test.

When you use IDT for Amazon IoT Greengrass to run the Amazon IoT Greengrass qualification suite, IDT performs the following steps:

  1. Loads and validates your device and credential configurations.

  2. Performs selected tests with the required local and cloud resources.

  3. Cleans up local and cloud resources.

  4. Generates tests reports that indicate if your device passed the tests required for qualification.

Test suite versions

IDT for Amazon IoT Greengrass organizes tests into test suites and test groups.

  • A test suite is the set of test groups used to verify that a device works with particular versions of Amazon IoT Greengrass.

  • A test group is the set of individual tests related to a particular feature, such as Greengrass group deployments and MQTT messaging.

Starting in IDT v3.0.0, test suites are versioned using a major.minor.patch format, for example GGQ_1.0.0. When you download IDT, the package includes the latest test suite version.

Important

IDT supports the three latest test suite versions for device qualification. For more information, see Support policy for Amazon IoT Device Tester for Amazon IoT Greengrass V1.

You can run list-supported-products to list the versions of Amazon IoT Greengrass and test suites that are supported by your current version of IDT. Tests from unsupported test suite versions are not valid for device qualification. IDT doesn't print qualification reports for unsupported versions.

Updates to IDT configuration settings

New tests might introduce new IDT configuration settings.

  • If the settings are optional, IDT continues running the tests.

  • If the settings are required, IDT notifies you and stops running. After you configure the settings, restart the test run.

    Configuration settings are located in the <device-tester-extract-location>/configs folder. For more information, see Configure IDT settings to run the Amazon IoT Greengrass qualification suite.

If an updated test suite version adds configuration settings, IDT creates a copy of the original configuration file in <device-tester-extract-location>/configs.

Test group descriptions

IDT v2.0.0 and later
Required Test Groups for Core Qualification

These test groups are required to qualify your Amazon IoT Greengrass device for the Amazon Partner Device Catalog.

Amazon IoT Greengrass Core Dependencies

Validates that your device meets all software and hardware requirements for the Amazon IoT Greengrass Core software.

The Software Packages Dependencies test case in this test group is not applicable when testing in a Docker container.

Deployment

Validates that Lambda functions can be deployed on your device.

MQTT

Verifies the Amazon IoT Greengrass message router functionality by checking local communication between the Greengrass core and client devices, which are local IoT devices.

Over-the-Air (OTA)

Validates that your device can successfully perform an OTA update of the Amazon IoT Greengrass Core software.

This test group is not applicable when testing in a Docker container.

Version

Checks that the version of Amazon IoT Greengrass provided is compatible with the Amazon IoT Device Tester version you are using.

Optional Test Groups

These test groups are optional. If you choose to qualify for optional tests, your device is listed with additional capabilities in the Amazon Partner Device Catalog.

Container Dependencies

Validates that the device meets all of the software and hardware requirements to run Lambda functions in container mode on a Greengrass core.

This test group is not applicable when testing in a Docker container.

Deployment Container

Validates that Lambda functions can be deployed on the device and run in container mode on a Greengrass core.

This test group is not applicable when testing in a Docker container.

Docker Dependencies (Supported for IDT v2.2.0 and later)

Validates that the device meets all the required technical dependencies to use the Greengrass Docker application deployment connector to run containers

This test group is not applicable when testing in a Docker container.

Hardware Security Integration (HSI)

Verifies that the provided HSI shared library can interface with the hardware security module (HSM) and implements the required PKCS#11 APIs correctly. The HSM and shared library must be able to sign a CSR, perform TLS operations, and provide the correct key lengths and public key algorithm.

Stream Manager Dependencies (Supported for IDT v2.2.0 and later)

Validates that the device meets all of the required technical dependencies to run Amazon IoT Greengrass stream manager.

Machine Learning Dependencies (Supported for IDT v3.1.0 and later)

Validates that the device meets all of the required technical dependencies to perform ML inference locally.

Machine Learning Inference Tests (Supported for IDT v3.1.0 and later)

Validates that ML inference can be performed on the given device under test. For more information, see Optional: Configuring your device for ML qualification.

Machine Learning Inference Container Tests (Supported for IDT v3.1.0 and later)

Validates that ML inference can be performed on the given device under test and run in container mode on a Greengrass core. For more information, see Optional: Configuring your device for ML qualification.

IDT v1.3.3 and earlier
Required Test Groups for Core Qualification

These tests are required to qualify your Amazon IoT Greengrass device for the Amazon Partner Device Catalog.

Amazon IoT Greengrass Core Dependencies

Validates that your device meets all software and hardware requirements for the Amazon IoT Greengrass Core software.

Combination (Device Security Interaction)

Verifies the functionality of the device certificate manager and IP detection on the Greengrass core device by changing connectivity information on the Greengrass group in the cloud. The test group rotates the Amazon IoT Greengrass server certificate and verifies that Amazon IoT Greengrass allows connections.

Deployment (Required for IDT v1.2 and earlier)

Validates that Lambda functions can be deployed on your device.

Device Certificate Manager (DCM)

Verifies that the Amazon IoT Greengrass device certificate manager can generate a server certificate on startup and rotate certificates if they are close to expiration.

IP Detection (IPD)

Verifies that core connectivity information is updated when there are IP address changes in a Greengrass core device. For more information, see Activate automatic IP detection.

Logging

Verifies that the Amazon IoT Greengrass logging service can write to a log file using a user Lambda function written in Python.

MQTT

Verifies the Amazon IoT Greengrass message router functionality by sending messages on a topic that is routed to two Lambda functions.

Native

Verifies that Amazon IoT Greengrass can run native (compiled) Lambda functions.

Over-the-Air (OTA)

Validates that your device can successfully perform a OTA update of the Amazon IoT Greengrass Core software.

Penetration

Validates that the Amazon IoT Greengrass Core software fails to start if hard link/soft link protection and seccomp are not enabled. It is also used to verify other security-related features.

Shadow

Verifies local shadow and shadow cloud-syncing functionality.

Spooler

Validates that the MQTT messages are queued with the default spooler configuration.

Token Exchange Service (TES)

Verifies that Amazon IoT Greengrass can exchange its core certificate for valid Amazon credentials.

Version

Checks that the version of Amazon IoT Greengrass provided is compatible with the Amazon IoT Device Tester version you are using.

Optional Test Groups

These tests are optional. If you choose to qualify for optional tests, your device is listed with additional capabilities in the Amazon Partner Device Catalog.

Container Dependencies

Checks that the device meets all of the required dependencies to run Lambda functions in container mode.

Hardware Security Integration (HSI)

Verifies that the provided HSI shared library can interface with the hardware security module (HSM) and implements the required PKCS#11 APIs correctly. The HSM and shared library must be able to sign a CSR, perform TLS operations, and provide the correct key lengths and public key algorithm.

Local Resource Access

Verifies the local resource access (LRA) feature of Amazon IoT Greengrass by providing access to local files and directories owned by various Linux users and groups to containerized Lambda functions through Amazon IoT Greengrass LRA APIs. Lambda functions should be allowed or denied access to local resources based on local resource access configuration.

Network

Verifies that socket connections can be established from a Lambda function. These socket connections should be allowed or denied based on the Greengrass core configuration.