Troubleshoot errors
Each test suite execution has a unique execution ID that is used to create a folder named
results/
in the
execution-id
results
directory. Individual test group logs are under the
results/
directory. Use the
IDT for FreeRTOS console output to find the execution id, test case id, and test group id of the
test case that failed and then open the log file for that test case named
execution-id
/logsresults/
.
The information in this file includes: execution-id
/logs/test_group_id
__test_case_id
.log
-
Full build and flash command output.
-
Test execution output.
-
More verbose IDT for FreeRTOS console output.
We recommend the following workflow for troubleshooting:
-
If you see the error "
user/role
is not authorized to access this resource", make sure that you configure permissions as specified in Create and configure an Amazon account. -
Read the console output to find information, such as execution UUID and currently executing tasks.
-
Look in the
FRQ_Report.xml
file for error statements from each test. This directory contains execution logs of each test group. -
Look in the log files under
/results/
.execution-id
/logs -
Investigate one of the following problem areas:
-
Device configuration, such as JSON configuration files in the
/configs/
folder. -
Device interface. Check the logs to determine which interface is failing.
-
Device tooling. Make sure that the toolchains for building and flashing the device are installed and configured correctly.
-
For FRQ 1.x.x make sure that a clean, cloned version of the FreeRTOS source code is available. FreeRTOS releases are tagged according to the FreeRTOS version. To clone a specific version of the code, use the following commands:
git clone --branch
version-number
https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout --init --recursive
-
Troubleshoot device configuration
When you use IDT for FreeRTOS, you must get the correct configuration files in place before
you execute the binary. If you're getting parsing and configuration errors, your first step
should be to locate and use a configuration template appropriate for your environment. These
templates are located in the
directory.IDT_ROOT
/configs
If you are still having issues, see the following debugging process.
Where do I look?
Start by reading the console output to find information, such as the execution UUID,
which is referenced as execution-id
in this documentation.
Next, look in the FRQ_Report.xml
file in the
/results/
directory. This
file contains all of the test cases that were run and error snippets for each failure. To
get all of the execution logs, look for the file
execution-id
/results/
for each test case.execution-id
/logs/test_group_id
__test_case_id
.log
IDT error codes
The following table explains the error codes generated by IDT for FreeRTOS:
Error Code | Error Code Name | Possible Root Cause | Troubleshooting |
---|---|---|---|
201 |
InvalidInputError |
Fields in |
Make sure required fields are not missing and are in required format in listed files. For more information, see First test of your microcontroller board. |
202 |
ValidationError |
Fields in |
Check the error message on the right hand side of the error code in the report:
|
203 |
CopySourceCodeError |
Unable to copy FreeRTOS source code to specified directory. |
Verify the following items:
|
204 |
BuildSourceError |
Unable to compile the FreeRTOS source code. |
Verify the following items:
|
205 |
FlashOrRunTestError |
IDT FreeRTOS is unable to flash or run FreeRTOS on your DUT. |
Verify the information under |
206 |
StartEchoServerError |
IDT FreeRTOS is unable to start echo server for the WiFi or secure sockets tests. |
Verify the ports configured under |
Debug config file parsing errors
Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quote from your JSON file. IDT for FreeRTOS performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you are still having issues locating the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.
Debug test results parse errors
While running a test group from
FreeRTOS-Libraries-Integration-Tests
In the above mentioned case, strange test case failure reasons like strings originating from unrelated device outputs are output. The IDT for FreeRTOS test case log file (which includes all the serial output IDT for FreeRTOS has received during the test) may show the following:
<unrelated device output> TEST(Full_PKCS11_Capabilities, PKCS11_Capabilities)<unrelated device output> <unrelated device output> PASS
In the above example, the unrelated device output prevents IDT for FreeRTOS from detecting the test result which is PASS.
Check the following to ensure optimal testing.
-
Make sure the logging macros used on the device are thread safe. See Implementing the library logging macros for more information.
-
Make sure there are minimal outputs to the serial connection during the tests. Other device outputs can be a problem even if your logging macros are properly thread safe, because the test results will output in separate calls during testing.
An IDT for FreeRTOS test case log would ideally show an uninterrupted test results output like below:
---------STARTING TESTS--------- TEST(Full_OTA_PAL, otaPal_CloseFile_ValidSignature) PASS TEST(Full_OTA_PAL, otaPal_CloseFile_InvalidSignatureBlockWritten) PASS ----------------------- 2 Tests 0 Failures 0 Ignored
Debug integrity check failures
If using FRQ 1.x.x version of FreeRTOS the following integrity checks apply.
When you run the FreeRTOSIntegrity test group and you encounter failures, first make
sure that you haven't modified any of the
directory files. If you haven’t,
and are still seeing issues, make sure you are using the correct branch. If you run IDT's
freertos
list-supported-products
command, you can find which tagged branch of the
repo you should be using.freertos
If you cloned the correct tagged branch of the freertos
repo and
still have issues, make sure you have also run the submodule update
command.
The clone workflow for the freertos
repo is as follows.
git clone --branch version-number https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout —init —recursive
The list of files the integrity checker looks for are in the
checksums.json
file in your
directory. To qualify a FreeRTOS
port without any modifications to files and the folder structure, make sure that none of the
files listed in the 'freertos
exhaustive
' and 'minimal
' sections of the
checksums.json
file have been modified. To run with an SDK
configured, verify that none of the files under the 'minimal
' section have been
modified.
If you run IDT with an SDK and have modified some files in your
directory, then make sure you
correctly configure your SDK in your freertos
userdata
file. Otherwise, the
Integrity checker will verify all files in the
directory.freertos
Debug FullWiFi test group failures
If you are using FRQ 1.x.x and encounter failures in the FullWiFi test group, and the
"AFQP_WiFiConnectMultipleAP
" test fails, this could be because both access
points aren't in the same subnet as the host computer running IDT. Make sure that both
access points are in the same subnet as the host computer running IDT.
Debug "required parameter missing" errors
Because new features are being added to IDT for FreeRTOS, changes to the configuration
files might be introduced. Using an old configuration file might break your configuration.
If this happens, the
file under the test_group_id
__test_case_id
.logresults/
directory explicitly lists all missing parameters. IDT for FreeRTOS validates your JSON
configuration file schemas to ensure that the latest supported version has been used.execution-id
/logs
Debug "test could not start" errors
You might see errors that point to failures during test start. Because there are several possible causes, check the following areas for correctness:
-
Make sure that the pool name you've included in your execution command actually exists. This is referenced directly from your
device.json
file. -
Make sure that the device or devices in your pool have correct configuration parameters.
Debug "unable to find start of test results" errors
You might see errors when IDT attempts to parse the results output by the device under test. There are several possible causes, so check the following areas for correctness:
-
Make sure that the device under test has a stable connection to your host machine. You can check the log file for a test that shows these errors to see what IDT is receiving.
-
If using FRQ 1.x.x, and the device under test is connected via a slow network or other interface, or you do not see the "---------STARTING TESTS---------" flag in a FreeRTOS test group log along with other FreeRTOS test group outputs, you can try increasing the value of
testStartDelayms
in your userdata configuration. For more information, see Configure build, flash, and test settings.
Debug a "Test failure:expected __ results but saw ___" errors
You might see errors that point to a test failure during testing.
The test expects a certain number of results, and does not see it during testing.
Some FreeRTOS tests run before IDT sees the output from the device.
If you see this error, you can try increasing the value of
testStartDelayms
in your userdata configuration.
For more information, see Configure build, flash, and test settings.
Debug a "________ was unselected due to ConditionalTests constraints" errors
This means that you are running a test on a device pool that is incompatible with the
test. This may happen with the OTA E2E tests. For example, while running the
OTADataplaneMQTT
test group and in your device.json
config file, you have chosen OTA as No or
OTADataPlaneProtocol
as HTTP. The test group
chosen to run must match your device.json
capability selections.
Debug an IDT timeout during device output monitoring
IDT can timeout due to a number of reasons. If a timeout happens during the device output monitoring phase of a test, and you can see the results inside of the IDT test case log, it means that the results were incorrectly parsed by IDT. One reason could be the interleaved log messages in the middle of the test results. If this is the case, please refer to the FreeRTOS Porting Guide for further details on how the UNITY logs should be setup.
Another reason for a timeout during device output monitoring could be a device rebooting after a single TLS test case failure. The device then runs the flashed image and causes an infinite loop which is seen in the logs. If this happens, make sure your device does not reboot after a test failure.
Debug a "not authorized to access resource" error
You might see the error "user/role
is not authorized to
access this resource" in the terminal output or in the test_manager.log
file under /results/
. To
resolve this issue, attach the execution-id
/logsAmazonIoTDeviceTesterForFreeRTOSFullAccess
managed
policy to your test user. For more information, see Create and configure an Amazon account.
Debug network test errors
For network-based tests, IDT starts an echo server that binds to a non-reserved port on the host machine. If you are running into errors due to timeouts or unavailable connections in the WiFi or secure sockets tests, make sure that your network is configured to allow traffic to configured ports in the 1024 - 49151 range.
The secure sockets test uses ports 33333 and 33334 by default. The WiFi tests uses port 33335 by default. If these three ports are in use or blocked by firewall or network, you can choose to use different ports in userdata.json for testing. For more information, see Configure build, flash, and test settings. You can use the following commands to check whether a specific port is in use:
-
Windows:
netsh advfirewall firewall show rule name=all | grep port
-
Linux:
sudo netstat -pan | grep port
-
macOS:
netstat -nat | grep port
OTA update failures due to same version payload
If OTA test cases are failing due to the same version being on the device after an OTA was performed, it may be due to your build system (e.g. cmake) not noticing IDT's changes to the FreeRTOS source code and not building an updated binary. This causes OTA to be performed with the same binary that is currently on the device, and the test to fail. To troubleshoot OTA update failures, start by making sure that you are using the latest supported version of your build system.
OTA test failure on PresignedUrlExpired
test
case
One prerequisite of this test is that the OTA update time should be more than 60 seconds, otherwise the test would fail. If this occurs, the following error message is found in the log: "Test takes less than 60 seconds (url expired time) to finish. Please reach out to us."
Debug device interface and port errors
This section contains information about the device interfaces IDT uses to connect to your devices.
Supported platforms
IDT supports Linux, macOS, and Windows. All three platforms have different naming schemes for serial devices that are attached to them:
-
Linux:
/dev/tty*
-
macOS:
/dev/tty.*
or/dev/cu.*
-
Windows: COM*
To check your device port:
-
For Linux/macOS, open a terminal and run
ls /dev/tty*
. -
For macOS, open a terminal and run
ls /dev/tty.*
orls /dev/cu.*
. -
For Windows, open Device Manager and expand the serial devices group.
To verify which device is connected to a port:
-
For Linux, make sure that the
udev
package is installed, and then runudevadm info –name=
. This utility prints the device driver information that helps you verify you are using the correct port.PORT
-
For macOS, open Launchpad and search for
System Information
. -
For Windows, open Device Manager and expand the serial devices group.
Device interfaces
Each embedded device is different, which means that they can have one or more serial ports. It is common for devices to have two ports when connected to a machine:
-
A data port for flashing the device.
-
A read port to read output.
You must set the correct read port in your
device.json
file. Otherwise, reading output from the device might fail.In the case of multiple ports, make sure to use the read port of the device in your
device.json
file. For example, if you plug in an Espressif WRover device and the two ports assigned to it are/dev/ttyUSB0
and/dev/ttyUSB1
, use/dev/ttyUSB1
in yourdevice.json
file.
For Windows, follow the same logic.
Reading device data
IDT for FreeRTOS uses individual device build and flash tooling to specify port configuration. If you are testing your device and don't get output, try the following default settings:
-
Baud rate: 115200
-
Data bits: 8
-
Parity: None
-
Stop bits: 1
-
Flow control: None
These settings are handled by IDT for FreeRTOS. You do not have to set them. However,
you can use the same method to manually read device output. On Linux or macOS, you can do
this with the screen
command. On Windows, you can use a program such as
TeraTerm.
Screen: screen /dev/cu.usbserial 115200
TeraTerm: Use the above-provided settings to set the fields explicitly in the
GUI.
Development toolchain problems
This section discusses problems that can occur with your toolchain.
Code Composer Studio on Ubuntu
Newer versions of Ubuntu (17.10 and 18.04) have a version of the glibc
package that is not compatible with Code Composer Studio 7.x
versions. We recommended that you install Code Composer Studio version 8.2 or
later.
Symptoms of incompatibility might include:
-
FreeRTOS failing to build or flash to your device.
-
The Code Composer Studio installer might freeze.
-
No log output is displayed in the console during the build or flash process.
-
Build command attempts to launch in GUI mode even when invoked as headless.
Logging
IDT for FreeRTOS logs are placed in a single location. From the root IDT directory, these
files are available under
results/
:execution-id
/
-
FRQ_Report.xml
-
awsiotdevicetester_report.xml
-
logs/
test_group_id
__test_case_id
.log
FRQ_Report.xml
and
logs/
are the most important logs to examine. test_group_id
__test_case_id
.logFRQ_Report.xml
contains
information about which test cases failed with a specific error message. You can then use
logs/
to dig further into the problem to get better context. test_group_id
__test_case_id
.log
Console errors
When Amazon IoT Device Tester is run, failures are reported to the console with brief messages. Look in
results/
to learn more about the error.execution-id
/logs/test_group_id
__test_case_id
.log
Log errors
Each test suite execution has a unique execution ID that is used to create a folder
named results/
. Individual
test case logs are under the
execution-id
results/
directory. Use
the output of the IDT for FreeRTOS console to find the execution id, test case id, and test
group id of the test case that failed. Then use this information to find and open the log
file for that test case named
execution-id
/logsresults/
The information in this file includes the full build and flash command output, test
execution output, and more verbose Amazon IoT Device Tester console output.execution-id
/logs/test_group_id
__test_case_id
.log
S3 bucket issues
If you press CTRL+C while running IDT, IDT will start a clean up process. Part of that clean up is to remove Amazon S3 resources that have been created as a part of the IDT tests. If the clean up can't finish, you might run into an issue where you have too many Amazon S3 buckets that have been created. This means the next time that you run IDT the tests will start to fail.
If you press CTRL+C to stop IDT, you must let it finish the clean up process to avoid this issue. You can also delete the Amazon S3 buckets from your account that were created manually.
Troubleshoot timeout errors
If you see timeout errors while running a test suite, increase the timeout by specifying a
timeout multiplier factor. This factor is applied to the default timeout value. Any value
configured for this flag must be greater than or equal to 1.0. To use the timeout multiplier,
use the flag --timeout-multiplier
when running the test suite.
Cellular feature and Amazon charges
When the Cellular
feature is set to Yes
in your
device.JSON
file, FullSecureSockets will use t.micro EC2 instances for
running tests and this may incur additional costs to your Amazon account. For more information,
see Amazon EC2 pricing
Qualification report generation policy
Qualification reports are only generated by Amazon IoT Device Tester (IDT) versions that support FreeRTOS
versions released within the last two years. If you have questions about the support policy,
please contact Amazon Web Services Support