Configure Amazon IoT Greengrass stream manager - Amazon IoT Greengrass
Stream manager parametersConfigure settings (console)Configure settings (CLI)See also
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Configure Amazon IoT Greengrass stream manager

On the Amazon IoT Greengrass core, stream manager can store, process, and export IoT device data. Stream manager provides parameters that you use to configure group-level runtime settings. These settings apply to all streams on the Greengrass core. You can use the Amazon IoT console or Amazon IoT Greengrass API to configure stream manager settings. Changes take effect after the group is deployed.

Note

After you configure stream manager, you can create and deploy IoT applications that run on the Greengrass core and interact with stream manager. These IoT applications are typically user-defined Lambda functions. For more information, see Use StreamManagerClient to work with streams.

Stream manager parameters

Stream manager provides the following parameters that allow you to define group-level settings. All parameters are optional.

Storage directory

Parameter name: STREAM_MANAGER_STORE_ROOT_DIR

The absolute path of the local directory used to store streams. This value must start with a forward slash (for example, /data).

For information about securing stream data, see Local data security.

Minimum Amazon IoT Greengrass Core version: 1.10.0

Server port

Parameter name: STREAM_MANAGER_SERVER_PORT

The local port number used to communicate with stream manager. The default is 8088.

Minimum Amazon IoT Greengrass Core version: 1.10.0

Authenticate client

Parameter name: STREAM_MANAGER_AUTHENTICATE_CLIENT

Indicates whether clients must be authenticated to interact with stream manager. All interaction between clients and stream manager is controlled by the Amazon IoT Greengrass Core SDK. This parameter determines which clients can call the Amazon IoT Greengrass Core SDK to work with streams. For more information, see Client authentication.

Valid values are true or false. The default is true (recommended).

  • true. Allows only Greengrass Lambda functions as clients. Lambda function clients use internal Amazon IoT Greengrass core protocols to authenticate with the Amazon IoT Greengrass Core SDK.

  • false. Allows any process that runs on the Amazon IoT Greengrass core to be a client. Do not set to false unless your business case requires it. For example, set this value to false only if non-Lambda processes on the core device must communicate directly with stream manager, such as Docker containers running on the core.

Minimum Amazon IoT Greengrass Core version: 1.10.0

Maximum bandwidth

Parameter name: STREAM_MANAGER_EXPORTER_MAX_BANDWIDTH

The average maximum bandwidth (in kilobits per second) that can be used to export data. The default allows unlimited use of available bandwidth.

Minimum Amazon IoT Greengrass Core version: 1.10.0

Thread pool size

Parameter name: STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE

The maximum number of active threads that can be used to export data. The default is 5.

The optimal size depends on your hardware, stream volume, and planned number of export streams. If your export speed is slow, you can adjust this setting to find the optimal size for your hardware and business case. The CPU and memory of your core device hardware are limiting factors. To start, you might try setting this value equal to the number of processor cores on the device.

Be careful not to set a size that's higher than your hardware can support. Each stream consumes hardware resources, so you should try to limit the number of export streams on constrained devices.

Minimum Amazon IoT Greengrass Core version: 1.10.0

JVM arguments

Parameter name: JVM_ARGS

Custom Java Virtual Machine arguments to pass to stream manager at startup. Multiple arguments should be separated by spaces.

Use this parameter only when you must override the default settings used by the JVM. For example, you might need to increase the default heap size if you plan to export a large number of streams.

Minimum Amazon IoT Greengrass Core version: 1.10.0

Read-only input file directories

Parameter name: STREAM_MANAGER_READ_ONLY_DIRS

A comma-separated list of absolute paths to the directories outside of the root file system that store input files. Stream manager reads and uploads the files to Amazon S3 and mounts the directories as read-only. For more information about exporting to Amazon S3, see Amazon S3 objects.

Use this parameter only if the following conditions are true:

  • The input file directory for a stream that exports to Amazon S3 is in one of the following locations:

    • A partition other than the root file system.

    • Under /tmp on the root file system.

  • The default containerization of the Greengrass group is Greengrass container.

Example value: /mnt/directory-1,/mnt/directory-2,/tmp

Minimum Amazon IoT Greengrass Core version: 1.11.0

Minimum size for multipart upload

Parameter name: STREAM_MANAGER_EXPORTER_S3_DESTINATION_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES

The minimum size (in bytes) of a part in a multipart upload to Amazon S3. Stream manager uses this setting and the size of the input file to determine how to batch data in a multipart PUT request. The default and minimum value is 5242880 bytes (5 MB).

Note

Stream manager uses the stream's sizeThresholdForMultipartUploadBytes property to determine whether to export to Amazon S3 as a single or multipart upload. User-defined Lambda functions set this threshold when they create a stream that exports to Amazon S3. The default threshold is 5 MB.

Minimum Amazon IoT Greengrass Core version: 1.11.0

Configure stream manager settings (console)

You can use the Amazon IoT console for the following management tasks:

Changes take effect after the Greengrass group is deployed. For a tutorial that shows how to deploy a Greengrass group that contains a Lambda function that interacts with stream manager, see Export data streams to the Amazon Web Services Cloud (console).

Note

When you use the console to enable stream manager and deploy the group, the memory size for stream manager is set to 4194304 KB (4 GB) by default. We recommend that you set the memory size to at least 128000 KB.

 

To check if stream manager is enabled (console)

  1. In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).

  2. Choose the target group.

  3. Choose the Lambda functions tab.

  4. Under System Lambda functions, select Stream manager, and choose Edit.

  5. Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.

 

To enable or disable stream manager during group creation (console)

  1. In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).

  2. Choose Create Group. Your choice on the next page determines how you configure stream manager for the group.

  3. Proceed through the Name your Group and choose a Greengrass core pages.

  4. Choose Create group.

  5. On the group configuration page, choose the Lambda functions tab, select Stream manager, and choose Edit.

    • To enable stream manager with default settings, choose Enable with default settings.

       

    • To enable stream manager with custom settings, choose Customize settings.

      1. On the Configure Stream manager page, choose Enable with custom settings.

      2. Under Custom settings, enter values for stream manager parameters. For more information, see Stream manager parameters. Leave fields empty to allow Amazon IoT Greengrass to use their default values.

         

    • To disable stream manager, choose Disable.

      1. On the Configure stream manager page, choose Disable.

         

  6. Choose Save.

  7. Continue through the remaining pages to create your group.

  8. On the Client devices page, download your security resources, review the information, and then choose Finish.

    Note

    When stream manager is enabled, you must install the Java 8 runtime on the core device before you deploy the group.

 

To enable or disable stream manager for an existing group (console)

  1. In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).

  2. Choose the target group.

  3. Choose the Lambda functions tab.

  4. Under System Lambda functions, select Stream manager, and choose Edit.

  5. Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.

 

To change stream manager settings (console)

  1. In the Amazon IoT console navigation pane, under Manage, expand Greengrass devices, and then choose Groups (V1).

  2. Choose the target group.

  3. Choose the Lambda functions tab.

  4. Under System Lambda functions, select Stream manager, and choose Edit.

  5. Check the enabled or disabled status. Any custom stream manager settings that are configured are also displayed.

  6. Choose Save.

Configure stream manager settings (CLI)

In the Amazon CLI, use the system GGStreamManager Lambda function to configure stream manager. System Lambda functions are components of the Amazon IoT Greengrass Core software. For stream manager and some other system Lambda functions, you can configure Greengrass functionality by managing the corresponding Function and FunctionDefinitionVersion objects in the Greengrass group. For more information, see Overview of the Amazon IoT Greengrass group object model.

You can use the API for the following management tasks. The examples in this section show how to use the Amazon CLI, but you can also call the Amazon IoT Greengrass API directly or use an Amazon SDK.

Changes take effect after the group is deployed. For a tutorial that shows how to deploy a Greengrass group with a Lambda function that interacts with stream manager, see Export data streams to the Amazon Web Services Cloud (CLI).

Tip

To see if stream manager is enabled and running from your core device, you can run the following command in a terminal on the device.

ps aux | grep -i 'streammanager'

 

To check if stream manager is enabled (CLI)

Stream manager is enabled if your deployed function definition version includes the system GGStreamManager Lambda function. To check, do the following;

  1. Get the IDs of the target Greengrass group and group version. This procedure assumes that this is the latest group and group version. The following query returns the most recently created group.

    aws greengrass list-groups --query "reverse(sort_by(Groups, &CreationTimestamp))[0]"

    Or, you can query by name. Group names are not required to be unique, so multiple groups might be returned.

    aws greengrass list-groups --query "Groups[?Name=='MyGroup']"
    Note

    You can also find these values in the Amazon IoT console. The group ID is displayed on the group's Settings page. Group version IDs are displayed on the group's Deployments tab.

  2. Copy the Id and LatestVersion values from the target group in the output.

  3. Get the latest group version.

    • Replace group-id with the Id that you copied.

    • Replace latest-group-version-id with the LatestVersion that you copied.

    aws greengrass get-group-version \
--group-id group-id \
--group-version-id latest-group-version-id

  4. From the FunctionDefinitionVersionArn in the output, get the IDs of the function definition and function definition version.

    • The function definition ID is the GUID that follows the functions segment in the Amazon Resource Name (ARN).

    • The function definition version ID is the GUID that follows the versions segment in the ARN.

    arn:aws:greengrass:us-west-2:123456789012:/greengrass/definition/functions/function-definition-id/versions/function-definition-version-id

  5. Get the function definition version.

    • Replace function-definition-id with the function definition ID.

    • Replace function-definition-version-id with the function definition version ID.

    aws greengrass get-function-definition-version \
--function-definition-id function-definition-id \
--function-definition-version-id function-definition-version-id

If the functions array in the output includes the GGStreamManager function, then stream manager is enabled. Any environment variables defined for the function represent custom settings for stream manager.

To enable, disable, or configure stream manager (CLI)

In the Amazon CLI, use the system GGStreamManager Lambda function to configure stream manager. Changes take effect after you deploy the group.

  • To enable stream manager, include GGStreamManager in the functions array of your function definition version. To configure custom settings, define environment variables for the corresponding stream manager parameters.

  • To disable stream manager, remove GGStreamManager from the functions array of your function definition version.

Stream manager with default settings

The following example configuration enables stream manager with default settings. It sets the arbitrary function ID to streamManager.

{
    "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1",
    "FunctionConfiguration": {
        "MemorySize": 4194304,
        "Pinned": true,
        "Timeout": 3
    },
    "Id": "streamManager"
}
Note

For the FunctionConfiguration properties, you might know the following:

  • MemorySize is set to 4194304 KB (4 GB) with default settings. You can always change this value. We recommend that you set MemorySize to at least 128000 KB.

  • Pinned must be set to true.

  • Timeout is required by the function definition version, but GGStreamManager doesn't use it.

Stream manager with custom settings

The following example configuration enables stream manager with custom values for the storage directory, server port, and thread pool size parameters.

{
    "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1",
    "FunctionConfiguration": {
        "Environment": {
            "Variables": {
                "STREAM_MANAGER_STORE_ROOT_DIR": "/data",
                "STREAM_MANAGER_SERVER_PORT": "1234",
                "STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE": "4"
            }
        },
        "MemorySize": 4194304,
        "Pinned": true,
        "Timeout": 3
    },
    "Id": "streamManager"
}

Amazon IoT Greengrass uses default values for stream manager parameters that aren't specified as environment variables.

Stream manager with custom settings for Amazon S3 exports

The following example configuration enables stream manager with custom values for the upload directory and minimum multipart upload size parameters.

{
    "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1",
    "FunctionConfiguration": {
        "Environment": {
            "Variables": {
                "STREAM_MANAGER_READ_ONLY_DIRS": "/mnt/directory-1,/mnt/directory-2,/tmp",
                "STREAM_MANAGER_EXPORTER_S3_DESTINATION_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES": "10485760"
            }
        },
        "MemorySize": 4194304,
        "Pinned": true,
        "Timeout": 3
    },
    "Id": "streamManager"
}

 

To enable, disable, or configure stream manager (CLI)

  1. Get the IDs of the target Greengrass group and group version. This procedure assumes that this is the latest group and group version. The following query returns the most recently created group.

    aws greengrass list-groups --query "reverse(sort_by(Groups, &CreationTimestamp))[0]"

    Or, you can query by name. Group names are not required to be unique, so multiple groups might be returned.

    aws greengrass list-groups --query "Groups[?Name=='MyGroup']"
    Note

    You can also find these values in the Amazon IoT console. The group ID is displayed on the group's Settings page. Group version IDs are displayed on the group's Deployments tab.

  2. Copy the Id and LatestVersion values from the target group in the output.

  3. Get the latest group version.

    • Replace group-id with the Id that you copied.

    • Replace latest-group-version-id with the LatestVersion that you copied.

    aws greengrass get-group-version \
--group-id group-id \
--group-version-id latest-group-version-id

  4. Copy the CoreDefinitionVersionArn and all other version ARNs from the output, except FunctionDefinitionVersionArn. You use these values later when you create a group version.

  5. From the FunctionDefinitionVersionArn in the output, copy the ID of the function definition. The ID is the GUID that follows the functions segment in the ARN, as shown in the following example.

    arn:aws:greengrass:us-west-2:123456789012:/greengrass/definition/functions/bcfc6b49-beb0-4396-b703-6dEXAMPLEcu5/versions/0f7337b4-922b-45c5-856f-1aEXAMPLEsf6
    Note

    Or, you can create a function definition by running the create-function-definition command, and then copying the ID from the output.

  6. Add a function definition version to the function definition.

    • Replace function-definition-id with the Id that you copied for the function definition.

    • In the functions array, include all other functions that you want to make available on the Greengrass core. You can use the get-function-definition-version command to get the list of existing functions.

     

    Enable stream manager with default settings

    The following example enables stream manager, by including the GGStreamManager function in the functions array. This example uses default values for stream manager parameters.

    aws greengrass create-function-definition-version \
--function-definition-id function-definition-id \
--functions '[
        {
            "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1",
            "FunctionConfiguration": {
                "MemorySize":  4194304,
                "Pinned": true,
                "Timeout": 3
            },
            "Id": "streamManager"
        },
        {    
            "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias",
            "FunctionConfiguration": {
                "Executable": "myLambdaFunction.function_handler",
                "MemorySize": 16000,
                "Pinned": true,
                "Timeout": 5
            },
            "Id": "myLambdaFunction"
        },
        ... more user-defined functions
    ]
}'
    Note

    The myLambdaFunction function in the examples represents one of your user-defined Lambda functions.

    Enable stream manager with custom settings

    The following example enables stream manager by including the GGStreamManager function in the functions array. All stream manager settings are optional, unless you want to change the default values. This example shows how to use environment variables to set custom values.

    aws greengrass create-function-definition-version \
--function-definition-id function-definition-id \
--functions '[
        {
            "FunctionArn": "arn:aws-cn:lambda:::function:GGStreamManager:1",
            "FunctionConfiguration": {
                "Environment": {
                    "Variables": {
                        "STREAM_MANAGER_STORE_ROOT_DIR": "/data",
                        "STREAM_MANAGER_SERVER_PORT": "1234",
                        "STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE": "4"
                    }
                },
                "MemorySize":  4194304,
                "Pinned": true,
                "Timeout": 3
            },
            "Id": "streamManager"
        },
        {    
            "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias",
            "FunctionConfiguration": {
                "Executable": "myLambdaFunction.function_handler",
                "MemorySize": 16000,
                "Pinned": true,
                "Timeout": 5
            },
            "Id": "myLambdaFunction"
        },
        ... more user-defined functions
    ]
}'
    Note

    For the FunctionConfiguration properties, you might know the following:

    • MemorySize is set to 4194304 KB (4 GB) with default settings. You can always change this value. We recommend that you set MemorySize to at least 128000 KB.

    • Pinned must be set to true.

    • Timeout is required by the function definition version, but GGStreamManager doesn't use it.

    Disable stream manager

    The following example omits the GGStreamManager function, which disables stream manager.

    aws greengrass create-function-definition-version \
--function-definition-id function-definition-id \
--functions '[
        {       
            "FunctionArn": "arn:aws-cn:lambda:us-west-2:123456789012:function:MyLambdaFunction:MyAlias",
            "FunctionConfiguration": {
                "Executable": "myLambdaFunction.function_handler",
                "MemorySize": 16000,
                "Pinned": true,
                "Timeout": 5
            },
            "Id": "myLambdaFunction"
        },
        ... more user-defined functions
    ]
}'
    Note

    If you don't want to deploy any Lambda functions, you can omit the function definition version entirely.

  7. Copy the Arn of the function definition version from the output.

  8. Create a group version that contains the system Lambda function.

    • Replace group-id with the Id for the group.

    • Replace core-definition-version-arn with the CoreDefinitionVersionArn that you copied from the latest group version.

    • Replace function-definition-version-arn with the Arn that you copied for the new function definition version.

    • Replace the ARNs for other group components (for example, SubscriptionDefinitionVersionArn or DeviceDefinitionVersionArn) that you copied from the latest group version.

    • Remove any unused parameters. For example, remove the --resource-definition-version-arn if your group version doesn't contain any resources.

    aws greengrass create-group-version \
--group-id group-id \
--core-definition-version-arn core-definition-version-arn \
--function-definition-version-arn function-definition-version-arn \
--device-definition-version-arn device-definition-version-arn \
--logger-definition-version-arn logger-definition-version-arn \
--resource-definition-version-arn resource-definition-version-arn \
--subscription-definition-version-arn subscription-definition-version-arn

  9. Copy the Version from the output. This is the ID of the new group version.

  10. Deploy the group with the new group version.

    • Replace group-id with the Id that you copied for the group.

    • Replace group-version-id with the Version that you copied for the new group version.

    aws greengrass create-deployment \
--group-id group-id \
--group-version-id group-version-id \
--deployment-type NewDeployment

 

Follow this procedure if you want to edit stream manager settings again later. Make sure to create a function definition version that includes the GGStreamManager function with the updated configuration. The group version must reference all component version ARNs that you want to deploy to the core. Changes take effect after the group is deployed.

See also