Splunk Integration connector - Amazon IoT Greengrass
RequirementsParametersInput dataOutput dataUsage ExampleLicensesChangelogSee also
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon IoT Greengrass Version 1 entered the extended life phase on June 30, 2023. For more information, see the Amazon IoT Greengrass V1 maintenance policy. After this date, Amazon IoT Greengrass V1 won't release updates that provide features, enhancements, bug fixes, or security patches. Devices that run on Amazon IoT Greengrass V1 won't be disrupted and will continue to operate and to connect to the cloud. We strongly recommend that you migrate to Amazon IoT Greengrass Version 2, which adds significant new features and support for additional platforms.

Splunk Integration connector

Warning

This connector has moved into the extended life phase, and Amazon IoT Greengrass won't release updates that provide features, enhancements to existing features, security patches, or bug fixes. For more information, see Amazon IoT Greengrass Version 1 maintenance policy.

The Splunk Integration connector publishes data from Greengrass devices to Splunk. This allows you to use Splunk to monitor and analyze the Greengrass core environment, and act on local events. The connector integrates with HTTP Event Collector (HEC). For more information, see Introduction to Splunk HTTP Event Collector in the Splunk documentation.

This connector receives logging and event data on an MQTT topic and publishes the data as is to the Splunk API.

You can use this connector to support industrial scenarios, such as:

  • Operators can use periodic data from actuators and sensors (for example, temperature, pressure, and water readings) to initiate alarms when values exceed certain thresholds.

  • Developers use data collected from industrial machinery to build ML models that can monitor the equipment for potential issues.

This connector has the following versions.

Version

ARN

4

arn:aws:greengrass:region::/connectors/SplunkIntegration/versions/4

3

arn:aws:greengrass:region::/connectors/SplunkIntegration/versions/3

2

arn:aws:greengrass:region::/connectors/SplunkIntegration/versions/2

1

arn:aws:greengrass:region::/connectors/SplunkIntegration/versions/1

For information about version changes, see the Changelog.

Requirements

This connector has the following requirements:

Version 3 - 4

  • Amazon IoT Greengrass Core software v1.9.3 or later. Amazon IoT Greengrass must be configured to support local secrets, as described in Secrets Requirements.

    Note

    This requirement includes allowing access to your Secrets Manager secrets. If you're using the default Greengrass service role, Greengrass has permission to get the values of secrets with names that start with greengrass-.

  • Python version 3.7 or 3.8 installed on the core device and added to the PATH environment variable.

    Note

    To use Python 3.8, run the following command to create a symbolic link from the the default Python 3.7 installation folder to the installed Python 3.8 binaries.

    sudo ln -s path-to-python-3.8/python3.8 /usr/bin/python3.7

    This configures your device to meet the Python requirement for Amazon IoT Greengrass.

  • The HTTP Event Collector functionality must be enabled in Splunk. For more information, see Set up and use HTTP eEvent Collector in Splunk Web in the Splunk documentation.

  • A text type secret in Amazon Secrets Manager that stores your Splunk HTTP Event Collector token. For more information, see About event collector tokens in the Splunk documentation and Creating a basic secret in the Amazon Secrets Manager User Guide.

    Note

    To create the secret in the Secrets Manager console, enter your token on the Plaintext tab. Don't include quotation marks or other formatting. In the API, specify the token as the value for the SecretString property.

  • A secret resource in the Greengrass group that references the Secrets Manager secret. For more information, see Deploy secrets to the Amazon IoT Greengrass core.

Versions 1 - 2

  • Amazon IoT Greengrass Core software v1.7 or later. Amazon IoT Greengrass must be configured to support local secrets, as described in Secrets Requirements.

    Note

    This requirement includes allowing access to your Secrets Manager secrets. If you're using the default Greengrass service role, Greengrass has permission to get the values of secrets with names that start with greengrass-.

  • Python version 2.7 installed on the core device and added to the PATH environment variable.

  • The HTTP Event Collector functionality must be enabled in Splunk. For more information, see Set up and use HTTP eEvent Collector in Splunk Web in the Splunk documentation.

  • A text type secret in Amazon Secrets Manager that stores your Splunk HTTP Event Collector token. For more information, see About event collector tokens in the Splunk documentation and Creating a basic secret in the Amazon Secrets Manager User Guide.

    Note

    To create the secret in the Secrets Manager console, enter your token on the Plaintext tab. Don't include quotation marks or other formatting. In the API, specify the token as the value for the SecretString property.

  • A secret resource in the Greengrass group that references the Secrets Manager secret. For more information, see Deploy secrets to the Amazon IoT Greengrass core.

Connector Parameters

This connector provides the following parameters:

Version 4
SplunkEndpoint

The endpoint of your Splunk instance. This value must contain the protocol, hostname, and port.

Display name in the Amazon IoT console: Splunk endpoint

Required: true

Type: string

Valid pattern: ^(http:\/\/|https:\/\/)?[a-z0-9]+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(\/.*)?$

MemorySize

The amount of memory (in KB) to allocate to the connector.

Display name in the Amazon IoT console: Memory size

Required: true

Type: string

Valid pattern: ^[0-9]+$

SplunkQueueSize

The maximum number of items to save in memory before the items are submitted or discarded. When this limit is met, the oldest items in the queue are replaced with newer items. This limit typically applies when there's no connection to the internet.

Display name in the Amazon IoT console: Maximum items to retain

Required: true

Type: string

Valid pattern: ^[0-9]+$

SplunkFlushIntervalSeconds

The interval (in seconds) for publishing received data to Splunk HEC. The maximum value is 900. To configure the connector to publish items as they are received (without batching), specify 0.

Display name in the Amazon IoT console: Splunk publish interval

Required: true

Type: string

Valid pattern: [0-9]|[1-9]\d|[1-9]\d\d|900

SplunkTokenSecretArn

The secret in Amazon Secrets Manager that stores the Splunk token. This must be a text type secret.

Display name in the Amazon IoT console: ARN of Splunk auth token secret

Required: true

Type: string

Valid pattern: arn:aws:secretsmanager:[a-z]{2}-[a-z]+-\d{1}:\d{12}?:secret:[a-zA-Z0-9-_]+-[a-zA-Z0-9-_]+

SplunkTokenSecretArn-ResourceId

The secret resource in the Greengrass group that references the Splunk secret.

Display name in the Amazon IoT console: Splunk auth token resource

Required: true

Type: string

Valid pattern: .+

SplunkCustomCALocation

The file path of the custom certificate authority (CA) for Splunk (for example, /etc/ssl/certs/splunk.crt).

Display name in the Amazon IoT console: Splunk custom certificate authority location

Required: false

Type: string

Valid pattern: ^$|/.*

IsolationMode

The containerization mode for this connector. The default is GreengrassContainer, which means that the connector runs in an isolated runtime environment inside the Amazon IoT Greengrass container.

Note

The default containerization setting for the group does not apply to connectors.

Display name in the Amazon IoT console: Container isolation mode

Required: false

Type: string

Valid values: GreengrassContainer or NoContainer

Valid pattern: ^NoContainer$|^GreengrassContainer$

Version 1 - 3
SplunkEndpoint

The endpoint of your Splunk instance. This value must contain the protocol, hostname, and port.

Display name in the Amazon IoT console: Splunk endpoint

Required: true

Type: string

Valid pattern: ^(http:\/\/|https:\/\/)?[a-z0-9]+([-.]{1}[a-z0-9]+)*.[a-z]{2,5}(:[0-9]{1,5})?(\/.*)?$

MemorySize

The amount of memory (in KB) to allocate to the connector.

Display name in the Amazon IoT console: Memory size

Required: true

Type: string

Valid pattern: ^[0-9]+$

SplunkQueueSize

The maximum number of items to save in memory before the items are submitted or discarded. When this limit is met, the oldest items in the queue are replaced with newer items. This limit typically applies when there's no connection to the internet.

Display name in the Amazon IoT console: Maximum items to retain

Required: true

Type: string

Valid pattern: ^[0-9]+$

SplunkFlushIntervalSeconds

The interval (in seconds) for publishing received data to Splunk HEC. The maximum value is 900. To configure the connector to publish items as they are received (without batching), specify 0.

Display name in the Amazon IoT console: Splunk publish interval

Required: true

Type: string

Valid pattern: [0-9]|[1-9]\d|[1-9]\d\d|900

SplunkTokenSecretArn

The secret in Amazon Secrets Manager that stores the Splunk token. This must be a text type secret.

Display name in the Amazon IoT console: ARN of Splunk auth token secret

Required: true

Type: string

Valid pattern: arn:aws:secretsmanager:[a-z]{2}-[a-z]+-\d{1}:\d{12}?:secret:[a-zA-Z0-9-_]+-[a-zA-Z0-9-_]+

SplunkTokenSecretArn-ResourceId

The secret resource in the Greengrass group that references the Splunk secret.

Display name in the Amazon IoT console: Splunk auth token resource

Required: true

Type: string

Valid pattern: .+

SplunkCustomCALocation

The file path of the custom certificate authority (CA) for Splunk (for example, /etc/ssl/certs/splunk.crt).

Display name in the Amazon IoT console: Splunk custom certificate authority location

Required: false

Type: string

Valid pattern: ^$|/.*

Create Connector Example (Amazon CLI)

The following CLI command creates a ConnectorDefinition with an initial version that contains the Splunk Integration connector.

aws greengrass create-connector-definition --name MyGreengrassConnectors --initial-version '{
    "Connectors": [
        {
            "Id": "MySplunkIntegrationConnector",
            "ConnectorArn": "arn:aws:greengrass:region::/connectors/SplunkIntegration/versions/4",
            "Parameters": {
                "SplunkEndpoint": "https://myinstance.cloud.splunk.com:8088",
                "MemorySize": 200000,
                "SplunkQueueSize": 10000,
                "SplunkFlushIntervalSeconds": 5,
                "SplunkTokenSecretArn":"arn:aws-cn:secretsmanager:region:account-id:secret:greengrass-secret-hash",
                "SplunkTokenSecretArn-ResourceId": "MySplunkResource", 
                "IsolationMode" : "GreengrassContainer"
            }
        }
    ]
}'
Note

The Lambda function in this connector has a long-lived lifecycle.

In the Amazon IoT Greengrass console, you can add a connector from the group's Connectors page. For more information, see Getting started with Greengrass connectors (console).

Input data

This connector accepts logging and event data on an MQTT topic and publishes the received data as is to the Splunk API. Input messages must be in JSON format.

Topic filter in subscription

splunk/logs/put

Message properties
request

The event data to send to the Splunk API. Events must meet the specifications of the services/collector API.

Required: true

Type: object. Only the event property is required.

id

An arbitrary ID for the request. This property is used to map an input request to an output status.

Required: false

Type: string

Limits

All limits that are imposed by the Splunk API apply when using this connector. For more information, see services/collector.

Example input
{
    "request": {
        "event": "some event",
        "fields": {
            "severity": "INFO",
            "category": [
                "value1",
                "value2"
            ]
        }
    },
    "id": "request123"
}

Output data

This connector publishes output data on two topics:

  • Status information on the splunk/logs/put/status topic.

  • Errors on the splunk/logs/put/error topic.

Topic filter: splunk/logs/put/status

Use this topic to listen for the status of the requests. Each time that the connector sends a batch of received data to the Splunk API, it publishes a list of the IDs of the requests that succeeded and failed.

Example output
{
    "response": {
        "succeeded": [
            "request123",
            ...
        ],
        "failed": [
            "request789",
            ...
        ]
    }
}
Topic filter: splunk/logs/put/error

Use this topic to listen for errors from the connector. The error_message property that describes the error or timeout encountered while processing the request.

Example output
{
    "response": {
        "error": "UnauthorizedException",
        "error_message": "invalid splunk token",
        "status": "fail"
    }
}
Note

If the connector detects a retryable error (for example, connection errors), it retries the publish in the next batch.

Usage Example

Use the following high-level steps to set up an example Python 3.7 Lambda function that you can use to try out the connector.

Note

  1. Make sure you meet the requirements for the connector.

  2. Create and publish a Lambda function that sends input data to the connector.

    Save the example code as a PY file. Download and unzip the Amazon IoT Greengrass Core SDK for Python. Then, create a zip package that contains the PY file and the greengrasssdk folder at the root level. This zip package is the deployment package that you upload to Amazon Lambda.

    After you create the Python 3.7 Lambda function, publish a function version and create an alias.

  3. Configure your Greengrass group.

    1. Add the Lambda function by its alias (recommended). Configure the Lambda lifecycle as long-lived (or "Pinned": true in the CLI).

    2. Add the required secret resource and grant read access to the Lambda function.

    3. Add the connector and configure its parameters.

    4. Add subscriptions that allow the connector to receive input data and send output data on supported topic filters.

      • Set the Lambda function as the source, the connector as the target, and use a supported input topic filter.

      • Set the connector as the source, Amazon IoT Core as the target, and use a supported output topic filter. You use this subscription to view status messages in the Amazon IoT console.

  4. Deploy the group.

  5. In the Amazon IoT console, on the Test page, subscribe to the output data topic to view status messages from the connector. The example Lambda function is long-lived and starts sending messages immediately after the group is deployed.

    When you're finished testing, you can set the Lambda lifecycle to on-demand (or "Pinned": false in the CLI) and deploy the group. This stops the function from sending messages.

Example

The following example Lambda function sends an input message to the connector.

import greengrasssdk
import time
import json

iot_client = greengrasssdk.client('iot-data')
send_topic = 'splunk/logs/put'

def create_request_with_all_fields():
    return {
        "request": {
            "event": "Access log test message."
        },
        "id" : "req_123"
    }

def publish_basic_message():
    messageToPublish = create_request_with_all_fields()
    print("Message To Publish: ", messageToPublish)
    iot_client.publish(topic=send_topic,
        payload=json.dumps(messageToPublish))

publish_basic_message()

def lambda_handler(event, context):
    return

Licenses

This connector is released under the Greengrass Core Software License Agreement.

Changelog

The following table describes the changes in each version of the connector.

Version

Changes

4

Added the IsolationMode parameter to configure the containerization mode for the connector.

3

Upgraded the Lambda runtime to Python 3.7, which changes the runtime requirement.

2

Fix to reduce excessive logging.

1

Initial release.

A Greengrass group can contain only one version of the connector at a time. For information about upgrading a connector version, see Upgrading connector versions.

See also