Review IDT test results and logs - Amazon IoT Greengrass
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Amazon IoT Greengrass Version 1 no longer receives feature updates, and will receive only security patches and bug fixes until June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Review IDT test results and logs

This section describes the format in which IDT generates console logs and test reports.

Console message format

Amazon IoT Device Tester uses a standard format for printing messages to the console when it starts a test suite. The following excerpt shows an example of a console message generated by IDT.

time="2000-01-02T03:04:05-07:00" level=info msg=Using suite: MyTestSuite_1.0.0 executionId=9a52f362-1227-11eb-86c9-8c8590419f30

Most console messages consist of the following fields:

time

A full ISO 8601 timestamp for the logged event.

level

The message level for the logged event. Typically, the logged message level is one of info, warn, or error. IDT issues a fatal or panic message if it encounters an expected event that causes it to exit early.

msg

The logged message.

executionId

A unique ID string for the current IDT process. This ID is used to differentiate between individual IDT runs.

Console messages generated from a test suite provide additional information about the device under test and the test suite, test group, and test cases that IDT runs. The following excerpt shows an example of a console message generated from a test suite.

time="2000-01-02T03:04:05-07:00" level=info msg=Hello world! suiteId=MyTestSuite groupId=myTestGroup testCaseId=myTestCase deviceId=my-device executionId=9a52f362-1227-11eb-86c9-8c8590419f30

The test-suite specific part of the console message contains the following fields:

suiteId

The name of the test suite currently running.

groupId

The ID of the test group currently running.

testCaseId

The ID of the test case current running.

deviceId

A ID of the device under test that the current test case is using.

To print a test summary to the console when a IDT finishes running a test, you must include a Report state in your state machine. The test summary contains information about the test suite, the test results for each group that was run, and the locations of the generated logs and report files. The following example shows a test summary message.

========== Test Summary ========== Execution Time: 5m00s Tests Completed: 4 Tests Passed: 3 Tests Failed: 1 Tests Skipped: 0 ---------------------------------- Test Groups: GroupA: PASSED GroupB: FAILED ---------------------------------- Failed Tests: Group Name: GroupB Test Name: TestB1 Reason: Something bad happened ---------------------------------- Path to IoT Device Tester Report: /path/to/awsiotdevicetester_report.xml Path to Test Execution Logs: /path/to/logs Path to Aggregated JUnit Report: /path/to/MyTestSuite_Report.xml

Amazon IoT Device Tester report schema

awsiotdevicetester_report.xml is a signed report that contains the following information:

  • The IDT version.

  • The test suite version.

  • The report signature and key used to sign the report.

  • The device SKU and the device pool name specified in the device.json file.

  • The product version and the device features that were tested.

  • The aggregate summary of test results. This information is the same as that contained in the suite-name_report.xml file.

<apnreport> <awsiotdevicetesterversion>idt-version</awsiotdevicetesterversion> <testsuiteversion>test-suite-version</testsuiteversion> <signature>signature</signature> <keyname>keyname</keyname> <session> <testsession>execution-id</testsession> <starttime>start-time</starttime> <endtime>end-time</endtime> </session> <awsproduct> <name>product-name</name> <version>product-version</version> <features> <feature name="<feature-name>" value="supported | not-supported | <feature-value>" type="optional | required"/> </features> </awsproduct> <device> <sku>device-sku</sku> <name>device-name</name> <features> <feature name="<feature-name>" value="<feature-value>"/> </features> <executionMethod>ssh | uart | docker</executionMethod> </device> <devenvironment> <os name="<os-name>"/> </devenvironment> <report> <suite-name-report-contents> </report> </apnreport>

The awsiotdevicetester_report.xml file contains an <awsproduct> tag that contains information about the product being tested and the product features that were validated after running a suite of tests.

Attributes used in the <awsproduct> tag

name

The name of the product being tested.

version

The version of the product being tested.

features

The features validated. Features marked as required are required for the test suite to validate the device. The following snippet shows how this information appears in the awsiotdevicetester_report.xml file.

<feature name="ssh" value="supported" type="required"></feature>

Features marked as optional are not required for validation. The following snippets show optional features.

<feature name="hsi" value="supported" type="optional"></feature>
 <feature name="mqtt" value="not-supported" type="optional"></feature>

Test suite report schema

The suite-name_Result.xml report is in JUnit XML format. You can integrate it into continuous integration and deployment platforms like Jenkins, Bamboo, and so on. The report contains an aggregate summary of test results.

<testsuites name="<suite-name> results" time="<run-duration>" tests="<number-of-test>" failures="<number-of-tests>" skipped="<number-of-tests>" errors="<number-of-tests>" disabled="0"> <testsuite name="<test-group-id>" package="" tests="<number-of-tests>" failures="<number-of-tests>" skipped="<number-of-tests>" errors="<number-of-tests>" disabled="0"> <!--success--> <testcase classname="<classname>" name="<name>" time="<run-duration>"/> <!--failure--> <testcase classname="<classname>" name="<name>" time="<run-duration>"> <failure type="<failure-type>"> reason </failure> </testcase> <!--skipped--> <testcase classname="<classname>" name="<name>" time="<run-duration>"> <skipped> reason </skipped> </testcase> <!--error--> <testcase classname="<classname>" name="<name>" time="<run-duration>"> <error> reason </error> </testcase> </testsuite> </testsuites>

The report section in both the awsiotdevicetester_report.xml or suite-name_report.xml lists the tests that were run and the results.

The first XML tag <testsuites> contains the summary of the test execution. For example:

<testsuites name="MyTestSuite results" time="2299" tests="28" failures="0" errors="0" disabled="0">

Attributes used in the <testsuites> tag

name

The name of the test suite.

time

The time, in seconds, it took to run the test suite.

tests

The number of tests executed.

failures

The number of tests that were run, but did not pass.

errors

The number of tests that IDT couldn't execute.

disabled

This attribute is not used and can be ignored.

In the event of test failures or errors, you can identify the test that failed by reviewing the <testsuites> XML tags. The <testsuite> XML tags inside the <testsuites> tag show the test result summary for a test group. For example:

<testsuite name="combination" package="" tests="1" failures="0" time="161" disabled="0" errors="0" skipped="0">

The format is similar to the <testsuites> tag, but with a skipped attribute that is not used and can be ignored. Inside each <testsuite> XML tag, there are <testcase> tags for each executed test for a test group. For example:

<testcase classname="Security Test" name="IP Change Tests" attempts="1"></testcase>>

Attributes used in the <testcase> tag

name

The name of the test.

attempts

The number of times IDT executed the test case.

When a test fails or an error occurs, <failure> or <error> tags are added to the <testcase> tag with information for troubleshooting. For example:

<testcase classname="mcu.Full_MQTT" name="MQTT_TestCase" attempts="1"> <failure type="Failure">Reason for the test failure</failure> <error>Reason for the test execution error</error> </testcase>