Stream manager
The stream manager component (aws.greengrass.StreamManager
) enables you to process data
streams to transfer to the Amazon Web Services Cloud from Greengrass core devices.
For more information about how to configure and use stream manager in custom components, see Manage data streams on Greengrass core devices.
Versions
This component has the following versions:
-
2.2.x
-
2.1.x
-
2.0.x
Note
If you use stream manager to export data to the cloud, you can't upgrade version 2.0.7 of the stream manager component to a version between v2.0.8 and v2.0.11. If you are deploying stream manager for the first time, we strongly recommend that you deploy the latest version of the stream manager component.
Type
This component is a generic component (aws.greengrass.generic
). The Greengrass nucleus runs the component's lifecycle
scripts.
For more information, see Component types.
Operating system
This component can be installed on core devices that run the following operating systems:
Linux
Windows
Requirements
This component has the following requirements:
-
The token exchange role must allow access to the Amazon Web Services Cloud destinations that you use with stream manager. For more information, see:
-
The stream manager component is supported to run in a VPC. To deploy this component in a VPC, the following is required.
-
The stream manager component must have connectivity to the Amazon service you publish data to.
-
Amazon S3:
com.amazonaws.
region
.s3 -
Amazon Kinesis Data Streams:
com.amazonaws.
region
.kinesis-streams -
Amazon IoT SiteWise:
com.amazonaws.
region
.iotsitewise.data
-
-
If you publish data to Amazon S3 in the
us-east-1
region, this component will attempt to use the S3 global endpoint by default; however, this endpoint is not available through the Amazon S3 VPC interface endpoint. For more information, see Restrictions and limitations of Amazon PrivateLink for Amazon S3. To resolve this, you can choose from the following options. -
Configure the stream manager component to use the regional S3 endpoint in the
us-east-1
region, by setting up-Daws.s3UseUsEast1RegionalEndpoint=regional
inJVM_ARGS
. -
Create an Amazon S3 gateway VPC endpoint instead of an Amazon S3 interface VPC endpoint. S3 gateway endpoints support access to the S3 global endpoint. For more information, see Create a gateway endpoint.
-
-
Endpoints and ports
This component must be able to perform outbound requests to the following endpoints and ports, in addition to endpoints and ports required for basic operation. For more information, see Allow device traffic through a proxy or firewall.
Endpoint | Port | Required | Description |
---|---|---|---|
|
443 | No |
Required if you publish data to Amazon IoT Analytics. |
|
443 | No |
Required if you publish data to Firehose. |
|
443 | No |
Required if you publish data to Amazon IoT SiteWise. |
|
443 | No |
Required if you publish data to S3 buckets. You can replace |
Dependencies
When you deploy a component, Amazon IoT Greengrass also deploys compatible versions of its dependencies. This means that you must meet the requirements for the component and all of its dependencies to successfully deploy the component. This section lists the dependencies for the released versions of this component and the semantic version constraints that define the component versions for each dependency. You can also view the dependencies for each version of the component in the Amazon IoT Greengrass console
For more information about component dependencies, see the component recipe reference.
Configuration
This component provides the following configuration parameters that you can customize when you deploy the component.
STREAM_MANAGER_STORE_ROOT_DIR
-
(Optional) The absolute path of the local directory used to store streams. This value must start with a forward slash (for example,
/data
).You must specify an existing folder, and the system user who runs the stream manager component must have permissions to read and write to this folder. For example, you can run the following commands to create and configure a folder,
/var/greengrass/streams
, which you specify as the stream manager root folder. These commands allow the default system user,ggc_user
, to read and write to this folder.sudo mkdir /var/greengrass/streams sudo chown ggc_user /var/greengrass/streams sudo chmod 700 /var/greengrass/streams
Default:
/work/aws.greengrass.StreamManager/greengrass/v2
STREAM_MANAGER_SERVER_PORT
-
(Optional) The local port number to use to communicate with stream manager.
You can specify
0
to use a random available port.Default:
8088
STREAM_MANAGER_AUTHENTICATE_CLIENT
-
(Optional) You can make it mandatory for clients to authenticate before they can interact with stream manager. The Stream Manager SDK controls interaction between clients and stream manager. This parameter determines which clients can call the Stream Manager SDK to work with streams. For more information, see stream manager client authentication.
If you specify
true
, the Stream Manager SDK allows only Greengrass components as clients.If you specify
false
, the Stream Manager SDK allows all processes on the core device to be clients.Default:
true
STREAM_MANAGER_EXPORTER_MAX_BANDWIDTH
-
(Optional) The average maximum bandwidth (in kilobits per second) that stream manager can use to export data.
Default: No limit
STREAM_MANAGER_EXPORTER_THREAD_POOL_SIZE
-
(Optional) The maximum number of active threads that stream manager can use to export data.
The optimal size depends on your hardware, stream volume, and planned number of export streams. If your export speed is slow, you can adjust this setting to find the optimal size for your hardware and business case. The CPU and memory of your core device hardware are limiting factors. To start, you might try setting this value equal to the number of processor cores on the device.
Be careful not to set a size that's higher than your hardware can support. Each stream consumes hardware resources, so try to limit the number of export streams on constrained devices.
Default: 5 threads
STREAM_MANAGER_EXPORTER_S3_DESTINATION_MULTIPART_UPLOAD_MIN_PART_SIZE_BYTES
-
(Optional) The minimum size (in bytes) of a part in a multipart upload to Amazon S3. Stream manager uses this setting and the size of the input file to determine how to batch data in a multipart PUT request.
Note
Stream manager uses the streams
sizeThresholdForMultipartUploadBytes
property to determine whether to export to Amazon S3 as a single or multipart upload. Amazon IoT Greengrass components can set this threshold when they create a stream that exports to Amazon S3.Default:
5242880
(5 MB). This is also the minimum value. LOG_LEVEL
-
(Optional) The logging level for the component. Choose from the following log levels, listed here in level order:
-
TRACE
-
DEBUG
-
INFO
-
WARN
-
ERROR
Default:
INFO
-
JVM_ARGS
-
(Optional) The custom Java Virtual Machine arguments to pass to stream manager at startup. Separate multiple arguments by spaces.
Use this parameter only when you must override the default settings used by the JVM. For example, you might need to increase the default heap size if you plan to export a large number of streams.
startupTimeoutSeconds
-
(Optional) The maximum of time in seconds for the component to start. The component's state changes to
ERRORED
if it exceeds this timeout.Default:
120
Example: Configuration merge update
The following example configuration specifies to use a non-default port.
{ "STREAM_MANAGER_SERVER_PORT": "18088" }
Local log file
This component uses the following log file.
To view this component's logs
-
Run the following command on the core device to view this component's log file in real time. Replace
or/greengrass/v2
C:\greengrass\v2
with the path to the Amazon IoT Greengrass root folder.
Changelog
The following table describes the changes in each version of the component.
Version |
Changes |
---|---|
2.2.0 |
|
2.1.13 |
|
2.1.12 |
|
2.1.11 |
Version updated for Greengrass nucleus version 2.12.0 release. |
2.1.10 |
|
2.1.9 |
Version updated for Greengrass nucleus version 2.11.0 release. |
2.1.8 |
|
2.1.7 |
|
2.1.6 |
|
2.1.5 |
Version updated for Greengrass nucleus version 2.10.0 release. |
2.1.4 |
|
2.1.3 |
|
2.1.2 |
|
2.1.1 |
Version updated for Greengrass nucleus version 2.8.0 release. |
2.1.0 |
|
2.0.15 |
Version updated for Greengrass nucleus version 2.6.0 release. |
2.0.14 |
This version contains bug fixes and improvements. |
2.0.13 |
Version updated for Greengrass nucleus version 2.5.0 release. |
2.0.12 |
|
2.0.11 |
Version updated for Greengrass nucleus version 2.4.0 release. |
2.0.10 |
Version updated for Greengrass nucleus version 2.3.0 release. |
2.0.9 |
Version updated for Greengrass nucleus version 2.2.0 release. |
2.0.8 |
Version updated for Greengrass nucleus version 2.1.0 release. |
2.0.7 |
Initial version. |