Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Manage data streams on the Amazon IoT Greengrass core

Amazon IoT Greengrass stream manager makes it easier and more reliable to transfer high-volume IoT data to the Amazon Web Services Cloud. Stream manager processes data streams locally and exports them to the Amazon Web Services Cloud automatically. This feature integrates with common edge scenarios, such as machine learning (ML) inference, where data is processed and analyzed locally before being exported to the Amazon Web Services Cloud or local storage destinations.

Stream manager simplifies application development. Your IoT applications can use a standardized mechanism to process high-volume streams and manage local data retention policies instead of building custom stream management functionality. IoT applications can read and write to streams. They can define policies for storage type, size, and data retention on a per-stream basis to control how stream manager processes and exports streams.

Stream manager is designed to work in environments with intermittent or limited connectivity. You can define bandwidth use, timeout behavior, and how stream data is handled when the core is connected or disconnected. For critical data, you can set priorities to control the order in which streams are exported to the Amazon Web Services Cloud.

You can configure automatic exports to the Amazon Web Services Cloud for storage or further processing and analysis. Stream manager supports exporting to the following Amazon Web Services Cloud destinations.

Stream management workflow

Your IoT applications interact with stream manager through the Amazon IoT Greengrass Core SDK. In a simple workflow, a user-defined Lambda function running on the Greengrass core consumes IoT data, such as time-series temperature and pressure metrics. The Lambda function might filter or compress the data and then call the Amazon IoT Greengrass Core SDK to write the data to a stream in stream manager. Stream manager can export the stream to the Amazon Web Services Cloud automatically, based on the policies defined for the stream. User-defined Lambda functions can also send data directly to local databases or storage repositories.

Your IoT applications can include multiple user-defined Lambda functions that read or write to streams. These local Lambda functions can read and write to streams to filter, aggregate, and analyze data locally. This makes it possible to respond quickly to local events and extract valuable information before the data is transferred from the core to cloud or local destinations.

An example workflow is shown in the following diagram.

To use stream manager, start by configuring stream manager parameters to define group-level runtime settings that apply to all streams on the Greengrass core. These customizable settings allow you to control how stream manager stores, processes, and exports streams based on your business need and environment constraints. For more information, see Configure Amazon IoT Greengrass stream manager.

After you configure stream manager, you can create and deploy your IoT applications. These are typically user-defined Lambda functions that use StreamManagerClient in the Amazon IoT Greengrass Core SDK to create and interact with streams. During stream creation, the Lambda function defines per-stream policies, such as export destinations, priority, and persistence. For more information, including code snippets for StreamManagerClient operations, see Use StreamManagerClient to work with streams.

For tutorials that configure a simple workflow, see Export data streams to the Amazon Web Services Cloud (console) or Export data streams to the Amazon Web Services Cloud (CLI).

Requirements

The following requirements apply for using stream manager:

Data security

When you use stream manager, be aware of the following security considerations.

Local data security

Amazon IoT Greengrass does not encrypt stream data at rest or in transit locally between components on the core device.

For more information, see Data encryption.

Client authentication

Stream manager clients use the Amazon IoT Greengrass Core SDK to communicate with stream manager. When client authentication is enabled, only Lambda functions in the Greengrass group can interact with streams in stream manager. When client authentication is disabled, any process running on the Greengrass core (such as Docker containers) can interact with streams in stream manager. You should disable authentication only if your business case requires it.

You use the STREAM_MANAGER_AUTHENTICATE_CLIENT parameter to set the client authentication mode. You can configure this parameter from the console or Amazon IoT Greengrass API. Changes take effect after the group is deployed.

See also