Tutorial: Use a confidence test from the confidence test suite
Amazon IoT Greengrass Testing Framework (GTF) and Greengrass Development Kit (GDK) offer developers ways to run end-to-end tests. You can complete this tutorial to initialize a GDK project with a component, initialize a GDK project with an end-to-end test module, and use a confidence test from the confidence test suite. After you build your custom test case, you can then run the test.
A confidence test is a generic test provided by Greengrass that validates fundamental component behaviors. These tests can be modified or extended to fit more specific component needs.
For this tutorial we will be using a HelloWorld component. If you are using another component, replace the HelloWorld component with your component.
In this tutorial, you do the following:
-
Initialize a GDK project with a component.
-
Initialize a GDK project with an end-to-end test module.
-
Use a test from the confidence test suite.
-
Add a tag to the new test case.
-
Build the test JAR.
-
Run the test.
Topics
Prerequisites
To complete this tutorial, you need the following:
-
GDK version 1.6.0 or later
-
Java
-
Maven
-
Git
Step 1: Initialize a GDK project with a component
-
Initialize an empty folder with a GDK project. Download the
HelloWorld
component implemented in Python by running the following command.gdk component init -t HelloWorld -l python -n HelloWorld
This command creates a new directory named
HelloWorld
in the current directory.
Step 2: Initialize a GDK project with an end-to-end test module
-
GDK enables you to download the testing module template consisting of a feature and step implementation. Run the following command to open the
HelloWorld
directory and initialize the existing GDK project using a testing module.cd HelloWorld gdk test-e2e init
This command creates a new directory named
gg-e2e-tests
within theHelloWorld
directory. This test directory is a Mavenproject which has a dependency on the Greengrass testing standalone JAR.
Step 3: Use a test from the confidence test suite
Writing a confidence test case consists of using the provided feature file and, if needed, modifying the scenarios. For an example of using a confidence test, see Example: Build a custom test case. Use the following steps to use a confidence test:
-
Use the provided feature file.
Navigate to
gg-e2e-tests/src/main/resources/greengrass/features
folder in the current directory. Open the sampleconfidenceTest.feature
file to use the confidence test.
Step 4: Add a tag to the new test case
-
You can assign tags to the features and scenarios to organize the test process. You can use tags to categorize the subsets of scenarios and also select hooks conditionally to run. Features and scenarios can have multiple tags separated by a space.
In this example, we are using the
HelloWorld
component.Each scenario is tagged with
@ConfidenceTest
. Change or add tags if you want to run only a subset of the test suite. Each test scenario is described at the top of each confidence test. The scenario is a series of steps that help with understanding the interactions and expected outcomes of each test case. You can extend these tests by adding your own steps or by modifying the existing ones.@ConfidenceTest Scenario: As a Developer, I can deploy GDK_COMPONENT_NAME to my device and see it is working as expected ....
Step 5: Build the test JAR
-
Build the component. You must build the component before building the test module.
gdk component build
-
Build the test module using the following command. This command will build the testing JAR in the
greengrass-build
folder.gdk test-e2e build
Step 6: Run the test
When you run a confidence test, the GTF automates the lifecycle of the test along with
managing resources that were created during the test. It first provisions a device under
test (DUT) as an Amazon IoT thing and installs the Greengrass core software on it. It will then create
a new component named HelloWorld
using the recipe specified in that path. The
HelloWorld
component is then deployed onto the core device through a Greengrass
thing deployment. It will then be verified if the deployment is successful. The deployment
status will changed to COMPLETED
within 3 minutes if the deployment is
successful.
-
Go to the
gdk-config.json
file in the project directory to target the tests with theConfidenceTest
tag or whichever tag yo8u specified in Step 4. Update the thetest-e2e
key using the following command."test-e2e":{ "gtf_options" : { "tags":"ConfidenceTest" } }
-
Before running the tests, you must provide Amazon credentials to the host device. GTF uses these credentials to manage the Amazon resources during the testing process. Make sure the role you provide has permissions to automate the necessary operations that are included in the test.
Run the following commands to provide the Amazon credentials.
-
-
Run the test using the following command.
gdk test-e2e run
This command downloads the latest version of the Greengrass nucleus in the
greengrass-build
folder and runs tests using it. This command also targets only the scenarios with theConfidenceTest
tag and generates a report for those scenarios. You will see the Amazon resources that were created during this test are discarded at the end of the test.
Example: Use a confidence test
The downloaded testing module in the GDK project consists of a provided feature file.
In the following example, we use a feature file for testing the thing deployment feature of the Greengrass software. We partially test the functionality of this feature with a scenario that performs deployment of a component through the Greengrass Amazon Web Services Cloud. This is a series of steps that help us to understand the interactions and expected outcomes of this use case.
-
Use the provided feature file.
Navigate to the
gg-e2e-tests/src/main/resources/greengrass/features
folder in the current directory. You can find the sampleconfidenceTest.feature
that looks like the following example.Feature: Confidence Test Suite Background: Given my device is registered as a Thing And my device is running Greengrass @ConfidenceTest Scenario: As a Developer, I can deploy GDK_COMPONENT_NAME to my device and see it is working as expected When I create a Greengrass deployment with components | GDK_COMPONENT_NAME | GDK_COMPONENT_RECIPE_FILE | | aws.greengrass.Cli | LATEST | And I deploy the Greengrass deployment configuration Then the Greengrass deployment is COMPLETED on the device after 180 seconds # Update component state accordingly. Possible states: {RUNNING, FINISHED, BROKEN, STOPPING} And I verify the GDK_COMPONENT_NAME component is
RUNNING
using the greengrass-cliEach test scenario is described at the top of each confidence test. The scenario is a series of steps that help with understanding the interactions and expected outcomes of each test case. You can extend these tests by adding your own steps or by modifying the existing ones. Each of the scenarios include comments that help you to make these adjustments.