MQTT
CONNECT, DISCONNECT, and RECONNECT
- "Device send CONNECT to Amazon IoT Core (Happy case)"
-
Validates that the device under test sends a CONNECT request.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes."tests":[ { "name":"
my_mqtt_connect_test
", "configuration": { // optional: "EXECUTION_TIMEOUT":"300
", // in seconds }, "test":{ "id":"MQTT_Connect", "version":"0.0.0" } } ] - "Device can return PUBACK to an arbitrary topic for QoS1"
-
This test case will check if the device (client) can return a PUBACK message if it received a publish message from the broker after subscribing to a topic with QoS1.
The payload content and the payload size are configurable for this test case. If the payload size is configured, Device Advisor will overwrite the value for the payload content, and send a predefined payload to the device with the desired size. The payload size is a value between 0 to 128 and cannot exceed 128 KB. Amazon IoT Core rejects publish and connect requests larger than 128 KB, as seen in the Amazon IoT Core message broker and protocol limits and quotas page.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes.PAYLOAD_SIZE
can be configured to a value between 0 and 128 kilobytes. Defining a payload size overrides the payload content as Device Advisor will be sending a pre-defined payload with the given size back to the device."tests":[ { "name":
"my_mqtt_client_puback_qos1"
, "configuration": { // optional:"TRIGGER_TOPIC": "myTopic", "EXECUTION_TIMEOUT":"300"
, // in seconds "PAYLOAD_FOR_PUBLISH_VALIDATION":"custom payload"
, "PAYLOAD_SIZE":"100"
// in kilobytes }, "test": { "id": "MQTT_Client_Puback_QoS1", "version": "0.0.0" } } ] - "Device connect retries with jitter backoff - No CONNACK response"
-
Validates that the device under test uses the proper jitter backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the device under test, and waits for the device under test to resend the request. The sixth connection attempt is allowed to pass through and CONNACK is allowed to flow back to the device under test.
The preceding process is performed again. In total, this test case requires the device to connect at least 12 times in total. The collected timestamps are used to validate that jitter backoff is used by the device under test. If the device under test has a strictly exponential backoff delay, this test case will pass with warnings.
We recommend implementation of the Exponential Backoff And Jitter
mechanism on the device under test to pass this test case. API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 4 minutes."tests":[ { "name":"my_mqtt_jitter_backoff_retries_test", "configuration": { // optional: "EXECUTION_TIMEOUT":
"300"
, // in seconds }, "test":{ "id":"MQTT_Connect_Jitter_Backoff_Retries", "version":"0.0.0" } } ] - "Device connect retries with exponential backoff - No CONNACK response"
-
Validates that the device under test uses the proper exponential backoff when reconnecting with the broker for at least five times. The broker logs the timestamp of the device under test's CONNECT request, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that an exponential backoff is used by the device under test.
We recommend implementation of the Exponential Backoff And Jitter
mechanism on the device under test to pass this test case. API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 4 minutes."tests":[ { "name":"
my_mqtt_exponential_backoff_retries_test
", "configuration": { // optional: "EXECUTION_TIMEOUT":"600
", // in seconds }, "test":{ "id":"MQTT_Connect_Exponential_Backoff_Retries", "version":"0.0.0" } } ] - "Device re-connect with jitter backoff - After server disconnect"
-
Validates if a device under test uses necessary jitter and backoff while reconnecting after it's been disconnected from the server. Device Advisor disconnects the device from the server for at least five times and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, pauses without sending a CONNACK to the client device, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail.
To pass this test case, we recommend implementing the Exponential Backoff And Jitter
mechanism on the device under test. API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 4 minutes.The number of reconnection attempts to validate for backoff can be changed by specifying the
RECONNECTION_ATTEMPTS
. The number must be between 5 and 10. The default value is 5."tests":[ { "name":"my_mqtt_reconnect_backoff_retries_on_server_disconnect", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "RECONNECTION_ATTEMPTS": 5 }, "test":{ "id":"MQTT_Reconnect_Backoff_Retries_On_Server_Disconnect", "version":"0.0.0" } } ]
- "Device re-connect with jitter backoff - On unstable connection"
-
Validates if a device under test uses necessary jitter and backoff while reconnecting on an unstable connection. Device Advisor disconnects the device from the server after five successful connections, and observes the device's behavior for MQTT reconnection. Device Advisor logs the timestamp of the CONNECT request for the device under test, performs packet validation, sends back CONNACK, disconnects, log the timestamp of the disconnection, and waits for the device under test to resend the request. The collected timestamps are used to validate that the device under test uses jitter and backoff while reconnecting after successful but unstable connections. If the device under test has a strictly exponential backoff or doesn't implement a proper jitter backoff mechanism, this test case will pass with warnings. If the device under test has implemented either a linear backoff or a constant backoff mechanism, the test will fail.
To pass this test case, we recommend implementing the Exponential Backoff And Jitter
mechanism on the device under test. API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 4 minutes.The number of reconnection attempts to validate for backoff can be changed by specifying the
RECONNECTION_ATTEMPTS
. The number must be between 5 and 10. The default value is 5."tests":[ { "name":"my_mqtt_reconnect_backoff_retries_on_unstable_connection", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300", // in seconds "RECONNECTION_ATTEMPTS": 5 }, "test":{ "id":"MQTT_Reconnect_Backoff_Retries_On_Unstable_Connection", "version":"0.0.0" } } ]
Publish
- "QoS0 (Happy Case)"
-
Validates that the device under test publishes a message with QoS0 or QoS1. You can also validate the topic of the message and payload by specifying the topic value and payload in the test settings.
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes."tests":[ { "name":"
my_mqtt_publish_test
", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300
", // in seconds "TOPIC_FOR_PUBLISH_VALIDATION":"my_TOPIC_FOR_PUBLISH_VALIDATION"
, "PAYLOAD_FOR_PUBLISH_VALIDATION":"my_PAYLOAD_FOR_PUBLISH_VALIDATION"
, }, "test":{ "id":"MQTT_Publish", "version":"0.0.0" } } ] - "QoS1 publish retry - No PUBACK"
-
Validates that the device under test republishes a message sent with QoS1, if the broker doesn't send PUBACK. You can also validate the topic of the message by specifying this topic in the test settings. The client device must not disconnect before republishing the message. This test also validates that the republished message has the same packet identifier as the original. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. It is recommended for at least 4 minutes."tests":[ { "name":"
my_mqtt_publish_retry_test
", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300
", // in seconds "TOPIC_FOR_PUBLISH_VALIDATION":"my_TOPIC_FOR_PUBLISH_VALIDATION"
, "PAYLOAD_FOR_PUBLISH_VALIDATION":"my_PAYLOAD_FOR_PUBLISH_VALIDATION"
, }, "test":{ "id":"MQTT_Publish_Retry_No_Puback", "version":"0.0.0" } } ] - "Publish Retained messages"
-
Validates that the device under test publishes a message with
retainFlag
set to true. You can validate the topic and payload of the message by setting the topic value and payload in the test settings. If theretainFlag
sent within the PUBLISH packet is not set to true, the test case will fail.API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes. To run this test case, add theiot:RetainPublish
action in your device role."tests":[ { "name":"
my_mqtt_publish_retained_messages_test
", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300
", // in seconds "TOPIC_FOR_PUBLISH_RETAINED_VALIDATION":"my_TOPIC_FOR_PUBLISH_RETAINED_VALIDATION"
, "PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION":"my_PAYLOAD_FOR_PUBLISH_RETAINED_VALIDATION"
, }, "test":{ "id":"MQTT_Publish_Retained_Messages", "version":"0.0.0" } } ] - "Publish with User Property"
-
Validates that the device under test publishes a message with the correct user property. You can validate the user property by setting the name-value pair in the test settings. If the user property is not provided or doesn't match, the test case fails.
API test case definition:
Note
This is a MQTT5 only test case.
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes."tests":[ { "name":"
my_mqtt_user_property_test
", "test":{ "USER_PROPERTIES": [ {"name": "name1", "value":"value1"}, {"name": "name2", "value":"value2"} ], "EXECUTION_TIMEOUT":"300
", // in seconds }, "test":{ "id":"MQTT_Publish_User_Property", "version":"0.0.0" } } ]
Subscribe
- "Can Subscribe (Happy Case)"
-
Validates that the device under test subscribes to MQTT topics. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 2 minutes."tests":[ { "name":"
my_mqtt_subscribe_test
", "configuration":{ // optional: "EXECUTION_TIMEOUT":"300
", // in seconds "TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["my_TOPIC_FOR_PUBLISH_VALIDATION_a"
,"my_TOPIC_FOR_PUBLISH_VALIDATION_b"
] }, "test":{ "id":"MQTT_Subscribe", "version":"0.0.0" } } ] - "Subscribe Retry - No SUBACK"
-
Validates that the device under test retries a failed subscription to MQTT topics. The server then waits and doesn't send a SUBACK. If the client device doesn't retry the subscription, the test fails. The client device must retry the failed subscription with the same packet Id. You can also validate the topic that the device under test subscribes to by specifying this topic in the test settings. During the test execution, if the device loses connection and reconnects, the test case will reset without failing and the device has to perform the test case steps again.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of 4 minutes."tests":[ { "name":"
my_mqtt_subscribe_retry_test
", "configuration":{ "EXECUTION_TIMEOUT":"300
", // in seconds // optional: "TOPIC_LIST_FOR_SUBSCRIPTION_VALIDATION":["myTOPIC_FOR_PUBLISH_VALIDATION_a"
,"my_TOPIC_FOR_PUBLISH_VALIDATION_b"
] }, "test":{ "id":"MQTT_Subscribe_Retry_No_Suback", "version":"0.0.0" } } ]
Keep-Alive
- "Mqtt No Ack PingResp"
-
This test case validates if the device under test disconnects when it doesn't receive a ping response. As part of this test case, Device Advisor blocks responses sent from Amazon IoT Core for publish, subscribe, and ping requests. It also validates if the device under test disconnects the MQTT connection.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout greater than 1.5 times thekeepAliveTime
value.The maximum
keepAliveTime
must be no greater than 230 seconds for this test."tests":[ { "name":"
Mqtt No Ack PingResp
", "configuration": //optional: "EXECUTION_TIMEOUT":"306
", // in seconds }, "test":{ "id":"MQTT_No_Ack_PingResp", "version":"0.0.0" } } ]
Persistent Session
- "Persistent Session (Happy Case)"
-
This test case validates the device behavior when disconnected from a persistent session. The test case checks if the device can reconnect, resume the subscriptions to its trigger topics without explicitly re-subscribing, receive the stored messages in the topics, and work as expected during a persistent session. When this test case passes, it indicates that the client device is able to maintain a persistent session with the Amazon IoT Core broker in an expected manner. For more information on Amazon IoT Persistent Sessions, see Using MQTT persistent sessions
. In this test case, the client device is expected to CONNECT with the Amazon IoT Core with a clean session flag set to false, and then subscribe to a trigger topic. After a successful subscription, the device will be disconnected by Amazon IoT Core Device Advisor. While the device is in a disconnected state, a QoS 1 message payload will be stored in that topic. Device Advisor will then allow the client device to re-connect with the test endpoint. At this point, since there is a persistent session, the client device is expected to resume its topic subscriptions without sending any additional SUBSCRIBE packets and receive the QoS 1 message from the broker. After re-connecting, if the client device re-subscribes to its trigger topic again by sending an additional SUBSCRIBE packet and/or if the client fails to receive the stored message from the trigger topic, the test case will fail.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. In the first connection, client device needs to explicitly subscribe to aTRIGGER_TOPIC
which was not subscribed before. To pass the test case, client device must successfully subscribe toTRIGGER_TOPIC
with a QoS 1. After re-connecting, the client device is expected to understand that there is an active persistent session; so it should accept the stored message sent by the trigger topic and return PUBACK for that specific message."tests":[ { "name":"my_mqtt_persistent_session_happy_case", "configuration":{ //required: "TRIGGER_TOPIC": "myTrigger/topic", // optional: // if Payload not provided, a string will be stored in the trigger topic to be sent back to the client device "PAYLOAD": "The message which should be received from AWS IoT Broker after re-connecting to a persistent session from the specified trigger topic.", "EXECUTION_TIMEOUT":"300" // in seconds }, "test":{ "id":"MQTT_Persistent_Session_Happy_Case", "version":"0.0.0" } } ]
- "Persistent Session - Session Expiry"
-
This test case helps to validate device behavior when a disconnected device reconnects to an expired persistent session. After the session expires, we expect the device to resubscribe to the topics previously subscribed to by explicitly sending a new SUBSCRIBE packet.
During the first connection, we expect the test device to CONNECT with the Amazon IoT broker, as its
CleanSession
flag is set to false to initiate a persistent session. The device should then subscribe to a trigger topic. Then the device is disconnected by Amazon IoT Core Device Advisor, after a successful subscription and initiation of a persistent session. After the disconnection, Amazon IoT Core Device Advisor allows the test device to re-connect back with the test endpoint. At this point, when the test device sends another CONNECT packet, Amazon IoT Core Device Advisor sends back a CONNACK packet that indicates that the persistent session is expired. The test device needs to interpret this packet properly, and it is expected to re-subscribe to the same trigger topic again as the persistent session is terminated. If the test device does not re-subscribe to its topic trigger again, the test case fails. For the test to pass, the device needs to understand that the persistent session is over, and send back a new SUBSCRIBE packet for the same trigger topic in the second connection.If this test case passes for a test device, it indicates that the device is able to handle re-connection on expiry of persistent session in an expected way.
API test case definition:
Note
EXECUTION_TIMEOUT
has a default value of 5 minutes. We recommend a timeout value of at least 4 minutes. The test device needs to explicitly subscribe to aTRIGGER_TOPIC
, to which it was not subscribed before. To pass the test case, the test device must send a CONNECT packet withCleanSession
flag set to false, and successfully subscribe to a trigger topic with a QoS 1. After a successful connection, Amazon IoT Core Device Advisor disconnects the device. After the disconnection, Amazon IoT Core Device Advisor allows the device to re-connect back, and the device is expected to re-subscribe to the sameTRIGGER_TOPIC
since Amazon IoT Core Device Advisor would have terminated the persistent session."tests":[ { "name":"my_expired_persistent_session_test", "configuration":{ //required: "TRIGGER_TOPIC": "myTrigger/topic", // optional: "EXECUTION_TIMEOUT":"300" // in seconds }, "test":{ "id":"MQTT_Expired_Persistent_Session", "version":"0.0.0" } } ]