Troubleshoot errors - FreeRTOS
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshoot errors

Each test suite execution has a unique execution ID that is used to create a folder named results/execution-id in the results directory. Individual test group logs are under the results/execution-id/logs directory. Use the IDT for FreeRTOS console output to find the execution id, test case id, and test group id of the test case that failed and then open the log file for that test case named results/execution-id/logs/test_group_id__test_case_id.log. The information in this file includes:

  • Full build and flash command output.

  • Test execution output.

  • More verbose IDT for FreeRTOS console output.

We recommend the following workflow for troubleshooting:

  1. If you see the error "user/role is not authorized to access this resource", make sure that you configure permissions as specified in Create and configure an Amazon account.

  2. Read the console output to find information, such as execution UUID and currently executing tasks.

  3. Look in the FRQ_Report.xml file for error statements from each test. This directory contains execution logs of each test group.

  4. Look in the log files under /results/execution-id/logs.

  5. Investigate one of the following problem areas:

    • Device configuration, such as JSON configuration files in the /configs/ folder.

    • Device interface. Check the logs to determine which interface is failing.

    • Device tooling. Make sure that the toolchains for building and flashing the device are installed and configured correctly.

    • For FRQ 1.x.x make sure that a clean, cloned version of the FreeRTOS source code is available. FreeRTOS releases are tagged according to the FreeRTOS version. To clone a specific version of the code, use the following commands:

      git clone --branch version-number https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout --init --recursive

Troubleshoot device configuration

When you use IDT for FreeRTOS, you must get the correct configuration files in place before you execute the binary. If you're getting parsing and configuration errors, your first step should be to locate and use a configuration template appropriate for your environment. These templates are located in the IDT_ROOT/configs directory.

If you are still having issues, see the following debugging process.

Where do I look?

Start by reading the console output to find information, such as the execution UUID, which is referenced as execution-id in this documentation.

Next, look in the FRQ_Report.xml file in the /results/execution-id directory. This file contains all of the test cases that were run and error snippets for each failure. To get all of the execution logs, look for the file /results/execution-id/logs/test_group_id__test_case_id.log for each test case.

IDT error codes

The following table explains the error codes generated by IDT for FreeRTOS:

Error Code Error Code Name Possible Root Cause Troubleshooting

201

InvalidInputError

Fields in device.json, config.json, or userdata.json are either missing or in an incorrect format.

Make sure required fields are not missing and are in required format in listed files. For more information, see First test of your microcontroller board.

202

ValidationError

Fields in device.json, config.json, or userdata.json contain invalid values.

Check the error message on the right hand side of the error code in the report:

  • Invalid Amazon Region - Specify a valid Amazon region in your config.json file. For more information about Amazon regions, see Regions and Endpoints.

  • Invalid Amazon credentials - Set valid Amazon credentials on your test machine (through environment variables or the credentials file). Verify that the authentication field is configured correctly. For more information, see Create and configure an Amazon account.

203

CopySourceCodeError

Unable to copy FreeRTOS source code to specified directory.

Verify the following items:

  • Check a valid sourcePath is specified in your userdata.json file.

  • Delete the build folder under FreeRTOS source code directory, if it exists. For more information, see Configure build, flash, and test settings.

  • Windows has a character limit for file path names. A long file path name will cause an error.

204

BuildSourceError

Unable to compile the FreeRTOS source code.

Verify the following items:

  • Check that the information under buildTool in your userdata.json file is correct.

  • If you are using cmake as a build tool, make sure the {{enableTests}} is specified in the buildTool command. For more information, see Configure build, flash, and test settings.

  • If you have extracted IDT for FreeRTOS to a file path on your system that contains spaces, for example C:\Users\My Name\Desktop\, you may need additional quotes inside of your build commands to make sure the paths are parsed properly. The same thing may be needed for your flash commands.

205

FlashOrRunTestError

IDT FreeRTOS is unable to flash or run FreeRTOS on your DUT.

Verify the information under flashTool in your userdata.json file is correct. For more information, see Configure build, flash, and test settings.

206

StartEchoServerError

IDT FreeRTOS is unable to start echo server for the WiFi or secure sockets tests.

Verify the ports configured under echoServerConfiguration in your userdata.json file are not in use or blocked by firewall or network settings.

Debug config file parsing errors

Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quote from your JSON file. IDT for FreeRTOS performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you are still having issues locating the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.

Debug test results parse errors

While running a test group from FreeRTOS-Libraries-Integration-Tests, such as FullTransportInterfaceTLS, FullPKCS11_Core, FullPKCS11_Onboard_ECC, FullPKCS11_Onboard_RSA, FullPKCS11_PreProvisioned_ECC, FullPKCS11_PreProvisioned_RSA, or OTACore, IDT for FreeRTOS parses the test results from the test device with the serial connection. Sometimes, extra serial outputs on the device may interfere with the parsing of the test results.

In the above mentioned case, strange test case failure reasons like strings originating from unrelated device outputs are output. The IDT for FreeRTOS test case log file (which includes all the serial output IDT for FreeRTOS has received during the test) may show the following:

<unrelated device output> TEST(Full_PKCS11_Capabilities, PKCS11_Capabilities)<unrelated device output> <unrelated device output> PASS

In the above example, the unrelated device output prevents IDT for FreeRTOS from detecting the test result which is PASS.

Check the following to ensure optimal testing.

  • Make sure the logging macros used on the device are thread safe. See Implementing the library logging macros for more information.

  • Make sure there are minimal outputs to the serial connection during the tests. Other device outputs can be a problem even if your logging macros are properly thread safe, because the test results will output in separate calls during testing.

An IDT for FreeRTOS test case log would ideally show an uninterrupted test results output like below:

---------STARTING TESTS--------- TEST(Full_OTA_PAL, otaPal_CloseFile_ValidSignature) PASS TEST(Full_OTA_PAL, otaPal_CloseFile_InvalidSignatureBlockWritten) PASS ----------------------- 2 Tests 0 Failures 0 Ignored

Debug integrity check failures

If using FRQ 1.x.x version of FreeRTOS the following integrity checks apply.

When you run the FreeRTOSIntegrity test group and you encounter failures, first make sure that you haven't modified any of the freertos directory files. If you haven’t, and are still seeing issues, make sure you are using the correct branch. If you run IDT's list-supported-products command, you can find which tagged branch of the freertos repo you should be using.

If you cloned the correct tagged branch of the freertos repo and still have issues, make sure you have also run the submodule update command. The clone workflow for the freertos repo is as follows.

git clone --branch version-number https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout —init —recursive

The list of files the integrity checker looks for are in the checksums.json file in your freertos directory. To qualify a FreeRTOS port without any modifications to files and the folder structure, make sure that none of the files listed in the 'exhaustive' and 'minimal' sections of the checksums.json file have been modified. To run with an SDK configured, verify that none of the files under the 'minimal' section have been modified.

If you run IDT with an SDK and have modified some files in your freertos directory, then make sure you correctly configure your SDK in your userdata file. Otherwise, the Integrity checker will verify all files in the freertos directory.

Debug FullWiFi test group failures

If you are using FRQ 1.x.x and encounter failures in the FullWiFi test group, and the "AFQP_WiFiConnectMultipleAP" test fails, this could be because both access points aren't in the same subnet as the host computer running IDT. Make sure that both access points are in the same subnet as the host computer running IDT.

Debug "required parameter missing" errors

Because new features are being added to IDT for FreeRTOS, changes to the configuration files might be introduced. Using an old configuration file might break your configuration. If this happens, the test_group_id__test_case_id.log file under the results/execution-id/logs directory explicitly lists all missing parameters. IDT for FreeRTOS validates your JSON configuration file schemas to ensure that the latest supported version has been used.

Debug "test could not start" errors

You might see errors that point to failures during test start. Because there are several possible causes, check the following areas for correctness:

  • Make sure that the pool name you've included in your execution command actually exists. This is referenced directly from your device.json file.

  • Make sure that the device or devices in your pool have correct configuration parameters.

Debug "unable to find start of test results" errors

You might see errors when IDT attempts to parse the results output by the device under test. There are several possible causes, so check the following areas for correctness:

  • Make sure that the device under test has a stable connection to your host machine. You can check the log file for a test that shows these errors to see what IDT is receiving.

  • If using FRQ 1.x.x, and the device under test is connected via a slow network or other interface, or you do not see the "---------STARTING TESTS---------" flag in a FreeRTOS test group log along with other FreeRTOS test group outputs, you can try increasing the value of testStartDelayms in your userdata configuration. For more information, see Configure build, flash, and test settings.

Debug a "Test failure:expected __ results but saw ___" errors

You might see errors that point to a test failure during testing. The test expects a certain number of results, and does not see it during testing. Some FreeRTOS tests run before IDT sees the output from the device. If you see this error, you can try increasing the value of testStartDelayms in your userdata configuration. For more information, see Configure build, flash, and test settings.

Debug a "________ was unselected due to ConditionalTests constraints" errors

This means that you are running a test on a device pool that is incompatible with the test. This may happen with the OTA E2E tests. For example, while running the OTADataplaneMQTT test group and in your device.json config file, you have chosen OTA as No or OTADataPlaneProtocol as HTTP. The test group chosen to run must match your device.json capability selections.

Debug an IDT timeout during device output monitoring

IDT can timeout due to a number of reasons. If a timeout happens during the device output monitoring phase of a test, and you can see the results inside of the IDT test case log, it means that the results were incorrectly parsed by IDT. One reason could be the interleaved log messages in the middle of the test results. If this is the case, please refer to the FreeRTOS Porting Guide for further details on how the UNITY logs should be setup.

Another reason for a timeout during device output monitoring could be a device rebooting after a single TLS test case failure. The device then runs the flashed image and causes an infinite loop which is seen in the logs. If this happens, make sure your device does not reboot after a test failure.

Debug a "not authorized to access resource" error

You might see the error "user/role is not authorized to access this resource" in the terminal output or in the test_manager.log file under /results/execution-id/logs. To resolve this issue, attach the AmazonIoTDeviceTesterForFreeRTOSFullAccess managed policy to your test user. For more information, see Create and configure an Amazon account.

Debug network test errors

For network-based tests, IDT starts an echo server that binds to a non-reserved port on the host machine. If you are running into errors due to timeouts or unavailable connections in the WiFi or secure sockets tests, make sure that your network is configured to allow traffic to configured ports in the 1024 - 49151 range.

The secure sockets test uses ports 33333 and 33334 by default. The WiFi tests uses port 33335 by default. If these three ports are in use or blocked by firewall or network, you can choose to use different ports in userdata.json for testing. For more information, see Configure build, flash, and test settings. You can use the following commands to check whether a specific port is in use:

  • Windows: netsh advfirewall firewall show rule name=all | grep port

  • Linux: sudo netstat -pan | grep port

  • macOS: netstat -nat | grep port

OTA update failures due to same version payload

If OTA test cases are failing due to the same version being on the device after an OTA was performed, it may be due to your build system (e.g. cmake) not noticing IDT's changes to the FreeRTOS source code and not building an updated binary. This causes OTA to be performed with the same binary that is currently on the device, and the test to fail. To troubleshoot OTA update failures, start by making sure that you are using the latest supported version of your build system.

OTA test failure on PresignedUrlExpired test case

One prerequisite of this test is that the OTA update time should be more than 60 seconds, otherwise the test would fail. If this occurs, the following error message is found in the log: "Test takes less than 60 seconds (url expired time) to finish. Please reach out to us."

Debug device interface and port errors

This section contains information about the device interfaces IDT uses to connect to your devices.

Supported platforms

IDT supports Linux, macOS, and Windows. All three platforms have different naming schemes for serial devices that are attached to them:

  • Linux: /dev/tty*

  • macOS: /dev/tty.* or /dev/cu.*

  • Windows: COM*

To check your device port:

  • For Linux/macOS, open a terminal and run ls /dev/tty*.

  • For macOS, open a terminal and run ls /dev/tty.* or ls /dev/cu.*.

  • For Windows, open Device Manager and expand the serial devices group.

To verify which device is connected to a port:

  • For Linux, make sure that the udev package is installed, and then run udevadm info –name=PORT. This utility prints the device driver information that helps you verify you are using the correct port.

  • For macOS, open Launchpad and search for System Information.

  • For Windows, open Device Manager and expand the serial devices group.

Device interfaces

Each embedded device is different, which means that they can have one or more serial ports. It is common for devices to have two ports when connected to a machine:

  • A data port for flashing the device.

  • A read port to read output.

    You must set the correct read port in your device.json file. Otherwise, reading output from the device might fail.

    In the case of multiple ports, make sure to use the read port of the device in your device.json file. For example, if you plug in an Espressif WRover device and the two ports assigned to it are /dev/ttyUSB0 and /dev/ttyUSB1, use /dev/ttyUSB1 in your device.json file.

For Windows, follow the same logic.

Reading device data

IDT for FreeRTOS uses individual device build and flash tooling to specify port configuration. If you are testing your device and don't get output, try the following default settings:

  • Baud rate: 115200

  • Data bits: 8

  • Parity: None

  • Stop bits: 1

  • Flow control: None

These settings are handled by IDT for FreeRTOS. You do not have to set them. However, you can use the same method to manually read device output. On Linux or macOS, you can do this with the screen command. On Windows, you can use a program such as TeraTerm.

Screen: screen /dev/cu.usbserial 115200

TeraTerm: Use the above-provided settings to set the fields explicitly in the GUI.

Development toolchain problems

This section discusses problems that can occur with your toolchain.

Code Composer Studio on Ubuntu

Newer versions of Ubuntu (17.10 and 18.04) have a version of the glibc package that is not compatible with Code Composer Studio 7.x versions. We recommended that you install Code Composer Studio version 8.2 or later.

Symptoms of incompatibility might include:

  • FreeRTOS failing to build or flash to your device.

  • The Code Composer Studio installer might freeze.

  • No log output is displayed in the console during the build or flash process.

  • Build command attempts to launch in GUI mode even when invoked as headless.

Logging

IDT for FreeRTOS logs are placed in a single location. From the root IDT directory, these files are available under results/execution-id/:

  • FRQ_Report.xml

  • awsiotdevicetester_report.xml

  • logs/test_group_id__test_case_id.log

FRQ_Report.xml and logs/test_group_id__test_case_id.log are the most important logs to examine. FRQ_Report.xml contains information about which test cases failed with a specific error message. You can then use logs/test_group_id__test_case_id.log to dig further into the problem to get better context.

Console errors

When Amazon IoT Device Tester is run, failures are reported to the console with brief messages. Look in results/execution-id/logs/test_group_id__test_case_id.log to learn more about the error.

Log errors

Each test suite execution has a unique execution ID that is used to create a folder named results/execution-id. Individual test case logs are under the results/execution-id/logs directory. Use the output of the IDT for FreeRTOS console to find the execution id, test case id, and test group id of the test case that failed. Then use this information to find and open the log file for that test case named results/execution-id/logs/test_group_id__test_case_id.log The information in this file includes the full build and flash command output, test execution output, and more verbose Amazon IoT Device Tester console output.

S3 bucket issues

If you press CTRL+C while running IDT, IDT will start a clean up process. Part of that clean up is to remove Amazon S3 resources that have been created as a part of the IDT tests. If the clean up can't finish, you might run into an issue where you have too many Amazon S3 buckets that have been created. This means the next time that you run IDT the tests will start to fail.

If you press CTRL+C to stop IDT, you must let it finish the clean up process to avoid this issue. You can also delete the Amazon S3 buckets from your account that were created manually.

Troubleshoot timeout errors

If you see timeout errors while running a test suite, increase the timeout by specifying a timeout multiplier factor. This factor is applied to the default timeout value. Any value configured for this flag must be greater than or equal to 1.0. To use the timeout multiplier, use the flag --timeout-multiplier when running the test suite.

IDT v3.0.0 and later
./devicetester_linux run-suite --suite-id FRQ_1.0.1 --pool-id DevicePool1 --timeout-multiplier 2.5
IDT v1.7.0 and earlier
./devicetester_linux run-suite --suite-id FRQ_1 --pool-id DevicePool1 --timeout-multiplier 2.5

Cellular feature and Amazon charges

When the Cellular feature is set to Yes in your device.JSON file, FullSecureSockets will use t.micro EC2 instances for running tests and this may incur additional costs to your Amazon account. For more information, see Amazon EC2 pricing.

Qualification report generation policy

Qualification reports are only generated by Amazon IoT Device Tester (IDT) versions that support FreeRTOS versions released within the last two years. If you have questions about the support policy, please contact Amazon Web Services Support.