Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.
Configure Amazon IoT Greengrass stream manager
On the Amazon IoT Greengrass core, stream manager can store, process, and export IoT device data. Stream manager provides parameters that you use to configure group-level runtime settings. These settings apply to all streams on the Greengrass core. You can use the Amazon IoT console or Amazon IoT Greengrass API to configure stream manager settings. Changes take effect after the group is deployed.
Note
After you configure stream manager, you can create and deploy IoT applications that run on the Greengrass core and interact with stream manager. These IoT applications are typically user-defined Lambda functions. For more information, see Use StreamManagerClient to work with streams.
Stream manager parameters
Stream manager provides the following parameters that allow you to define group-level settings. All parameters are optional.
- Storage directory
-
Parameter name:
STREAM_MANAGER_STORE_ROOT_DIR
The absolute path of the local directory used to store streams. This value must start with a forward slash (for example,
/data
).For information about securing stream data, see Local data security.
Minimum Amazon IoT Greengrass Core version: 1.10.0
- Server port
-
Parameter name:
STREAM_MANAGER_SERVER_PORT
The local port number used to communicate with stream manager. The default is
8088
.Minimum Amazon IoT Greengrass Core version: 1.10.0
- Authenticate client
-
Parameter name:
STREAM_MANAGER_AUTHENTICATE_CLIENT
Indicates whether clients must be authenticated to interact with stream manager. All interaction between clients and stream manager is controlled by the Amazon IoT Greengrass Core SDK. This parameter determines which clients can call the Amazon IoT Greengrass Core SDK to work with streams. For more information, see Client authentication.
Valid values are
true
orfalse
. The default istrue
(recommended).-
true
. Allows only Greengrass Lambda functions as clients. Lambda function clients use internal Amazon IoT Greengrass core protocols to authenticate with the Amazon IoT Greengrass Core SDK. -
false
. Allows any process that runs on the Amazon IoT Greengrass core to be a client. Do not set tofalse
unless your business case requires it. For example, set this value tofalse
only if non-Lambda processes on the core device must communicate directly with stream manager, such as Docker containers running on the core.
Minimum Amazon IoT Greengrass Core version: 1.10.0
-
- Maximum bandwidth
-
Parameter name:
STREAM_MANAGER_EXPORTER_MAX_BANDWIDTH
The average maximum bandwidth (in kilobits per second) that can be used to export data. The default allows unlimited use of available bandwidth.
Minimum Amazon IoT Greengrass Core version: 1.10.0
- Thread pool size
-
Parameter name:
STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE
The maximum number of active threads that can be used to export data. The default is
5
.The optimal size depends on your hardware, stream volume, and planned number of export streams. If your export speed is slow, you can adjust this setting to find the optimal size for your hardware and business case. The CPU and memory of your core device hardware are limiting factors. To start, you might try setting this value equal to the number of processor cores on the device.
Be careful not to set a size that's higher than your hardware can support. Each stream consumes hardware resources, so you should try to limit the number of export streams on constrained devices.
Minimum Amazon IoT Greengrass Core version: 1.10.0
- JVM arguments
-
Parameter name:
JVM_ARGS
Custom Java Virtual Machine arguments to pass to stream manager at startup. Multiple arguments should be separated by spaces.
Use this parameter only when you must override the default settings used by the JVM. For example, you might need to increase the default heap size if you plan to export a large number of streams.
Minimum Amazon IoT Greengrass Core version: 1.10.0
- Read-only input file directories
-
Parameter name:
STREAM_MANAGER_READ_ONLY_DIRS
A comma-separated list of absolute paths to the directories outside of the root file system that store input files. Stream manager reads and uploads the files to Amazon S3 and mounts the directories as read-only. For more information about exporting to Amazon S3, see Amazon S3 objects.
Use this parameter only if the following conditions are true:
-
The input file directory for a stream that exports to Amazon S3 is in one of the following locations:
-
A partition other than the root file system.
-
Under
/tmp
on the root file system.
-
-
The default containerization of the Greengrass group is Greengrass container.
Example value:
/mnt/directory-1,/mnt/directory-2,/tmp
Minimum Amazon IoT Greengrass Core version: 1.11.0
-
- Minimum size for multipart upload
-
Parameter name:
STREAM_MANAGER_EXPORTER_S3_DESTINATION_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES
The minimum size (in bytes) of a part in a multipart upload to Amazon S3. Stream manager uses this setting and the size of the input file to determine how to batch data in a multipart PUT request. The default and minimum value is
5242880
bytes (5 MB).Note
Stream manager uses the stream's
sizeThresholdForMultipartUploadBytes
property to determine whether to export to Amazon S3 as a single or multipart upload. User-defined Lambda functions set this threshold when they create a stream that exports to Amazon S3. The default threshold is 5 MB.Minimum Amazon IoT Greengrass Core version: 1.11.0
Configure stream manager settings (console)
You can use the Amazon IoT console for the following management tasks:
Changes take effect after the Greengrass group is deployed. For a tutorial that shows how to deploy a Greengrass group that contains a Lambda function that interacts with stream manager, see Export data streams to the Amazon Web Services Cloud (console).
Note
When you use the console to enable stream manager and deploy the group, the memory size for stream manager is set to 4194304 KB (4 GB) by default. We recommend that you set the memory size to at least 128000 KB.
To check if stream manager is enabled (console)
In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).
Choose the target group.
-
Choose the Lambda functions tab.
-
Under System Lambda functions, select Stream manager, and choose Edit.
-
Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.
To enable or disable stream manager during group creation (console)
In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).
-
Choose Create Group. Your choice on the next page determines how you configure stream manager for the group.
-
Proceed through the Name your Group and choose a Greengrass core pages.
-
Choose Create group.
-
On the group configuration page, choose the Lambda functions tab, select Stream manager, and choose Edit.
-
To enable stream manager with default settings, choose Enable with default settings.
-
To enable stream manager with custom settings, choose Customize settings.
-
On the Configure Stream manager page, choose Enable with custom settings.
-
Under Custom settings, enter values for stream manager parameters. For more information, see Stream manager parameters. Leave fields empty to allow Amazon IoT Greengrass to use their default values.
-
-
To disable stream manager, choose Disable.
-
On the Configure stream manager page, choose Disable.
-
-
-
Choose Save.
-
Continue through the remaining pages to create your group.
-
On the Client devices page, download your security resources, review the information, and then choose Finish.
Note
When stream manager is enabled, you must install the Java 8 runtime on the core device before you deploy the group.
To enable or disable stream manager for an existing group (console)
In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).
Choose the target group.
-
Choose the Lambda functions tab.
-
Under System Lambda functions, select Stream manager, and choose Edit.
-
Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.
To change stream manager settings (console)
In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).
Choose the target group.
-
Choose the Lambda functions tab.
-
Under System Lambda functions, select Stream manager, and choose Edit.
-
Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.
-
Choose Save.
Configure stream manager settings (CLI)
In the Amazon CLI, use the system GGStreamManager
Lambda function to configure
stream manager. System Lambda functions are components of the Amazon IoT Greengrass Core software. For
stream manager and some other system Lambda functions, you can configure Greengrass
functionality by managing the corresponding Function
and
FunctionDefinitionVersion
objects in the Greengrass group. For more
information, see Overview of the Amazon IoT Greengrass group object model.
You can use the API for the following management tasks. The examples in this section show how to use the Amazon CLI, but you can also call the Amazon IoT Greengrass API directly or use an Amazon SDK.
Changes take effect after the group is deployed. For a tutorial that shows how to deploy a Greengrass group with a Lambda function that interacts with stream manager, see Export data streams to the Amazon Web Services Cloud (CLI).
Tip
To see if stream manager is enabled and running from your core device, you can run the following command in a terminal on the device.
ps aux | grep -i 'streammanager'
To check if stream manager is enabled (CLI)
Stream manager is enabled if your deployed function definition version includes
the system GGStreamManager
Lambda function. To check, do the
following;
-
Get the IDs of the target Greengrass group and group version. This procedure assumes that this is the latest group and group version. The following query returns the most recently created group.
aws greengrass list-groups --query "reverse(sort_by(Groups, &CreationTimestamp))[0]"
Or, you can query by name. Group names are not required to be unique, so multiple groups might be returned.
aws greengrass list-groups --query "Groups[?Name=='
MyGroup
']"Note
You can also find these values in the Amazon IoT console. The group ID is displayed on the group's Settings page. Group version IDs are displayed on the group's Deployments tab.
-
Copy the
Id
andLatestVersion
values from the target group in the output. -
Get the latest group version.
-
Replace
group-id
with theId
that you copied. -
Replace
latest-group-version-id
with theLatestVersion
that you copied.
aws greengrass get-group-version \ --group-id
group-id
\ --group-version-idlatest-group-version-id
-
-
From the
FunctionDefinitionVersionArn
in the output, get the IDs of the function definition and function definition version.-
The function definition ID is the GUID that follows the
functions
segment in the Amazon Resource Name (ARN). -
The function definition version ID is the GUID that follows the
versions
segment in the ARN.
arn:aws:greengrass:us-west-2:123456789012:/greengrass/definition/functions/
function-definition-id
/versions/function-definition-version-id
-
-
Get the function definition version.
-
Replace
function-definition-id
with the function definition ID. -
Replace
function-definition-version-id
with the function definition version ID.
aws greengrass get-function-definition-version \ --function-definition-id
function-definition-id
\ --function-definition-version-idfunction-definition-version-id
-
If the functions
array in the output includes the
GGStreamManager
function, then stream manager is enabled. Any
environment variables defined for the function represent custom settings for stream
manager.
To enable, disable, or configure stream manager (CLI)
In the Amazon CLI, use the system GGStreamManager
Lambda function to
configure stream manager. Changes take effect after you deploy the group.
-
To enable stream manager, include
GGStreamManager
in thefunctions
array of your function definition version. To configure custom settings, define environment variables for the corresponding stream manager parameters. -
To disable stream manager, remove
GGStreamManager
from thefunctions
array of your function definition version.
- Stream manager with default settings
-
The following example configuration enables stream manager with default settings. It sets the arbitrary function ID to
streamManager
.{ "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1", "FunctionConfiguration": { "MemorySize": 4194304, "Pinned": true, "Timeout": 3 }, "Id": "streamManager" }
Note
For the
FunctionConfiguration
properties, you might know the following:-
MemorySize
is set to 4194304 KB (4 GB) with default settings. You can always change this value. We recommend that you setMemorySize
to at least 128000 KB. -
Pinned
must be set totrue
. -
Timeout
is required by the function definition version, butGGStreamManager
doesn't use it.
-
- Stream manager with custom settings
-
The following example configuration enables stream manager with custom values for the storage directory, server port, and thread pool size parameters.
{ "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1", "FunctionConfiguration": { "Environment": { "Variables": { "STREAM_MANAGER_STORE_ROOT_DIR": "/data", "STREAM_MANAGER_SERVER_PORT": "1234", "STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE": "4" } }, "MemorySize": 4194304, "Pinned": true, "Timeout": 3 }, "Id": "streamManager" }
Amazon IoT Greengrass uses default values for stream manager parameters that aren't specified as environment variables.
- Stream manager with custom settings for Amazon S3 exports
-
The following example configuration enables stream manager with custom values for the upload directory and minimum multipart upload size parameters.
{ "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1", "FunctionConfiguration": { "Environment": { "Variables": { "STREAM_MANAGER_READ_ONLY_DIRS": "/mnt/directory-1,/mnt/directory-2,/tmp", "STREAM_MANAGER_EXPORTER_S3_DESTINATION_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES": "10485760" } }, "MemorySize": 4194304, "Pinned": true, "Timeout": 3 }, "Id": "streamManager" }
To enable, disable, or configure stream manager (CLI)
-
Get the IDs of the target Greengrass group and group version. This procedure assumes that this is the latest group and group version. The following query returns the most recently created group.
aws greengrass list-groups --query "reverse(sort_by(Groups, &CreationTimestamp))[0]"
Or, you can query by name. Group names are not required to be unique, so multiple groups might be returned.
aws greengrass list-groups --query "Groups[?Name=='
MyGroup
']"Note
You can also find these values in the Amazon IoT console. The group ID is displayed on the group's Settings page. Group version IDs are displayed on the group's Deployments tab.
-
Copy the
Id
andLatestVersion
values from the target group in the output. -
Get the latest group version.
-
Replace
group-id
with theId
that you copied. -
Replace
latest-group-version-id
with theLatestVersion
that you copied.
aws greengrass get-group-version \ --group-id
group-id
\ --group-version-idlatest-group-version-id
-
-
Copy the
CoreDefinitionVersionArn
and all other version ARNs from the output, exceptFunctionDefinitionVersionArn
. You use these values later when you create a group version. -
From the
FunctionDefinitionVersionArn
in the output, copy the ID of the function definition. The ID is the GUID that follows thefunctions
segment in the ARN, as shown in the following example.arn:aws:greengrass:us-west-2:123456789012:/greengrass/definition/functions/bcfc6b49-beb0-4396-b703-6dEXAMPLEcu5/versions/0f7337b4-922b-45c5-856f-1aEXAMPLEsf6
Note
Or, you can create a function definition by running the
create-function-definition
command, and then copying the ID from the output. -
Add a function definition version to the function definition.
-
Replace
function-definition-id
with theId
that you copied for the function definition. -
In the
functions
array, include all other functions that you want to make available on the Greengrass core. You can use theget-function-definition-version
command to get the list of existing functions.
- Enable stream manager with default settings
-
The following example enables stream manager, by including the
GGStreamManager
function in thefunctions
array. This example uses default values for stream manager parameters.aws greengrass create-function-definition-version \ --function-definition-id
function-definition-id
\ --functions '[ { "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1", "FunctionConfiguration": { "MemorySize": 4194304, "Pinned": true, "Timeout": 3 }, "Id": "streamManager" }, { "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias", "FunctionConfiguration": { "Executable": "myLambdaFunction.function_handler", "MemorySize": 16000, "Pinned": true, "Timeout": 5 }, "Id": "myLambdaFunction" }, ... more user-defined functions ] }'Note
The
myLambdaFunction
function in the examples represents one of your user-defined Lambda functions. - Enable stream manager with custom settings
-
The following example enables stream manager by including the
GGStreamManager
function in thefunctions
array. All stream manager settings are optional, unless you want to change the default values. This example shows how to use environment variables to set custom values.aws greengrass create-function-definition-version \ --function-definition-id
function-definition-id
\ --functions '[ { "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1", "FunctionConfiguration": { "Environment": { "Variables": { "STREAM_MANAGER_STORE_ROOT_DIR": "/data", "STREAM_MANAGER_SERVER_PORT": "1234", "STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE": "4" } }, "MemorySize": 4194304, "Pinned": true, "Timeout": 3 }, "Id": "streamManager" }, { "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias", "FunctionConfiguration": { "Executable": "myLambdaFunction.function_handler", "MemorySize": 16000, "Pinned": true, "Timeout": 5 }, "Id": "myLambdaFunction" }, ... more user-defined functions ] }'Note
For the
FunctionConfiguration
properties, you might know the following:-
MemorySize
is set to 4194304 KB (4 GB) with default settings. You can always change this value. We recommend that you setMemorySize
to at least 128000 KB. -
Pinned
must be set totrue
. -
Timeout
is required by the function definition version, butGGStreamManager
doesn't use it.
-
- Disable stream manager
-
The following example omits the
GGStreamManager
function, which disables stream manager.aws greengrass create-function-definition-version \ --function-definition-id
function-definition-id
\ --functions '[ { "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias", "FunctionConfiguration": { "Executable": "myLambdaFunction.function_handler", "MemorySize": 16000, "Pinned": true, "Timeout": 5 }, "Id": "myLambdaFunction" }, ... more user-defined functions ] }'Note
If you don't want to deploy any Lambda functions, you can omit the function definition version entirely.
-
-
Copy the
Arn
of the function definition version from the output. -
Create a group version that contains the system Lambda function.
-
Replace
group-id
with theId
for the group. -
Replace
core-definition-version-arn
with theCoreDefinitionVersionArn
that you copied from the latest group version. -
Replace
function-definition-version-arn
with theArn
that you copied for the new function definition version. -
Replace the ARNs for other group components (for example,
SubscriptionDefinitionVersionArn
orDeviceDefinitionVersionArn
) that you copied from the latest group version. -
Remove any unused parameters. For example, remove the
--resource-definition-version-arn
if your group version doesn't contain any resources.
aws greengrass create-group-version \ --group-id
group-id
\ --core-definition-version-arncore-definition-version-arn
\ --function-definition-version-arnfunction-definition-version-arn
\ --device-definition-version-arndevice-definition-version-arn
\ --logger-definition-version-arnlogger-definition-version-arn
\ --resource-definition-version-arnresource-definition-version-arn
\ --subscription-definition-version-arnsubscription-definition-version-arn
-
-
Copy the
Version
from the output. This is the ID of the new group version. -
Deploy the group with the new group version.
-
Replace
group-id
with theId
that you copied for the group. -
Replace
group-version-id
with theVersion
that you copied for the new group version.
aws greengrass create-deployment \ --group-id
group-id
\ --group-version-idgroup-version-id
\ --deployment-type NewDeployment -
Follow this procedure if you want to edit stream manager settings again later.
Make sure to create a function definition version that includes the
GGStreamManager
function with the updated configuration. The group
version must reference all component version ARNs that you want to deploy to the
core. Changes take effect after the group is deployed.