Amazon IoT Device Advisor troubleshooting guide - Amazon IoT Core
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Device Advisor troubleshooting guide

Help us improve this topic

Let us know what would help make it better

General
Q: Can I run multiple test suites in parallel?

A: Yes. Device Advisor now supports running multiple test suites on different devices using a Device-level endpoint. If you use the Account-level endpoint, you can run one suite at a time because one Device Advisor endpoint is available per account. For more information see Configure your device.

Q: I saw from my device that the TLS connection was denied by Device Advisor. Is this expected?

A: Yes. Device Advisor denies the TLS connection before and after each test run. We recommend that users implement a device retry mechanism to have a fully automated testing experience with Device Advisor. If you execute a test suite with more than one test case, for example TLS connect, MQTT connect, and MQTT publish, then we recommend that you have a mechanism built for your device. The mechanism can try to connect to our test endpoint every 5 seconds for a minute to two. In this way you can run multiple test cases in sequence in an automated manner.

Q: Can I get a history of Device Advisor API calls made on my account for security analysis and operational troubleshooting purposes?

A: Yes. To receive a history of Device Advisor API calls made on your account, you simply turn on CloudTrail in the Amazon IoT Management Console and filter the event source to be iotdeviceadvisor.amazonaws.com.

Q: How do I view Device Advisor logs in CloudWatch?

A: Logs generated during a test suite run are uploaded to CloudWatch if you add the required policy (for example, CloudWatchFullAccess) to your service role (see Setting up). If there is at least one test case in the test suite, a log group "aws/iot/deviceadvisor/$testSuiteId" is created with two log streams. One stream is the "$testRunId" and includes logs of actions taken before and after executing the test cases in your test suite, such as setup and cleanup steps. The other log stream is "$suiteRunId_$testRunId," which is specific to a test suite run. Events sent from devices and Amazon IoT Core will be logged to this log stream.

Q: What is the purpose of the device permission role?

A: Device Advisor stands between your test device and Amazon IoT Core to simulate test scenarios. It accepts connections and messages from your test devices and forwards them to Amazon IoT Core by assuming your device permission role and initiating a connection on your behalf. It’s important to make sure the device role permissions are the same as those on the certificate you use for running tests. Amazon IoT certificate policies are not enforced when Device Advisor initiates a connection to Amazon IoT Core on your behalf by using the device permission role. However, the permissions from the device permission role you set are enforced.

Q: In what Regions is Device Advisor supported?

A: Device Advisor is supported in us-east-1, us-west-2, ap-northeast-1, and eu-west-1 Regions.

Q: Why do I see inconsistent results?

A: One of the primary causes of inconsistent results is setting a test's EXECUTION_TIMEOUT to a value that is too low. For more information about recommended and default EXECUTION_TIMEOUT values, see Device Advisor test cases.

Q: What MQTT protocol does Device Advisor support?

A: Device Advisor supports MQTT Version 3.1.1 with X509 client certificates.

Q: What if my test case failed with an execution timed out message even though I tried to connect my device to the test endpoint?

A: Validate all the steps under Create an IAM role to be used as your device role. If the test still fails, it could be that the device is not sending the correct Server Name Indication (SNI) extension, which is required for Device Advisor to work. The correct SNI value is the endpoint address returned when following the Configure your device section. Amazon IoT also requires devices to send the Server Name Indication (SNI) extension to the Transport Layer Security (TLS) protocol. For more information, see Transport security in Amazon IoT.

Q: My MQTT connection fails with an "libaws-c-mqtt: AWS_ERROR_MQTT_UNEXPECTED_HANGUP" error (or) my device’s MQTT connection is being automatically disconnected from the Device Advisor endpoint. How can this error be resolved?

A: This particular error code and unexpected disconnections can be caused by many different things, but is most likely related to the device role attached to the device. The below checkpoints (in order of priority) will resolve this issue.

  • The device role attached to the device must have the minimum IAM permissions required to run the tests. Device Advisor will use the attached device role to perform Amazon IoT MQTT actions on behalf of the test device. If required permissions are absent, then the AWS_ERROR_MQTT_UNEXPECTED_HANGUP error will be seen or unexpected disconnections will happen while the device tries to connect to Device Advisor endpoint. For example, if you selected to run the MQTT Publish test case, both Connect and Publish actions must be included in the role with the corresponding ClientId and Topic (you can provide multiple values by using commas to separate the values, and you can provide prefix values using a wildcard (*) character. For example: To provide permissions to publish on any topic beginning with TestTopic, you can provide "TestTopic*” as the resource value. Here are some examples of policies.

  • Mismatch between the values defined in the device role for your resource types and the actual values used in code. For example: A mismatch in ClientId defined in the role and the actual ClientId used in your device code. Values like ClientId, Topic, and TopicFilter must be identical in the device role and code.

  • The device certificate attached to your device must be active and have a policy attached to it with the required action permissions for resources. Note that, the device certificate policy grants or denies access to Amazon IoT resources and Amazon IoT Core data plane operations. Device Advisor requires you to have an active device certificate attached to your device which grants the action permissions used during a test case.