Step 2: Create a Kinesis Data Firehose Delivery Stream with Splunk as a Destination

To create a Kinesis Data Firehose delivery stream with Splunk as a destination

1. Open the Kinesis Data Firehose console at https://console.amazonaws.cn/firehose/.

2. Choose Create delivery stream.

3. For the name of the delivery stream, enter VPCtoSplunkStream. Then scroll to the bottom, and choose Next.

4. For Data transformation*, choose Enabled.

5. For Lambda function*, choose Create new.

6. In the Choose Lambda blueprint pane, scroll down and choose Kinesis Firehose Cloudwatch Logs Processor. This opens the Amazon Lambda console.

   Note

   It is recommended that you choose the low Lambda buffering hint value of 256 KB.

   

7. On the Amazon Lambda console, for the function name, enter VPCtoSplunkLambda.

8. In the description text under Execution role, choose the IAM console link to create a custom role. This opens the Amazon Identity and Access Management (IAM) console.

9. In the IAM console, choose Lambda.

10. Choose Next: Permissions.

11. Choose Create policy.

12. Choose the JSON tab and replace the existing JSON with the following. Be sure to replace the your-region and your-aws-account-id placeholders with your Amazon Region code and account ID. Don't include any hyphens or dashes in the account ID. For a list of Amazon Region codes, see Amazon Regions and Endpoints.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "firehose:PutRecordBatch"
                ],
                "Resource": [
                    "arn:aws:firehose:your-region:your-aws-account-id:deliverystream/VPCtoSplunkStream"
                ]
            }
        ]
    }

    This policy allows the Lambda function to put data back into the delivery stream by invoking the PutRecordBatch operation. This step is needed because a Lambda function can only return up to 6 MiB of data every time Kinesis Data Firehose invokes it. If the size of the uncompressed data exceeds 6 MiB, the function invokes PutRecordBatch to put some of the data back into the delivery stream for future processing.

13. Back in the Create role window, refresh the list of policies, then choose VPCtoSplunkLambdaPolicy by selecting the box to its left.

14. Choose Next: Tags.

15. Choose Next: Review.

16. For Role Name, enter VPCtoSplunkLambdaRole, then choose Create role.

17. Back in the Lambda console, refresh the list of existing roles, then select VPCtoSplunkLambdaRole.

18. Scroll down and choose Create function.

19. In the Lambda function pane, scroll down to the Basic settings section, and increase the timeout to 3 minutes.

20. Scroll up and choose Save.

21. Back in the Choose Lambda blueprint dialog box, choose Close.

22. On the delivery stream creation page, under the Transform source records with Amazon Lambda section, choose the refresh button. Then choose VPCtoSplunkLambda in the list of functions.

23. Scroll down and choose Next.

24. For Destination*, choose Splunk.

25. For Splunk cluster endpoint, see the information at Configure Amazon Kinesis Firehose to send data to the Splunk platform in the Splunk documentation.

26. Keep Splunk endpoint type set to Raw endpoint.

27. Enter the value (and not the name) of your Splunk HTTP Event Collector (HEC) token.

28. For S3 backup mode*, choose Backup all events.

29. Choose an existing Amazon S3 bucket (or create a new one if you want), and choose Next.

30. On the Configure settings page, scroll down to the IAM role section, and choose Create new or choose.

31. In the IAM role list, choose Create a new IAM role. For Role Name, enter VPCtoSplunkLambdaFirehoseRole, and then choose Allow.

32. Choose Next, and review the configuration that you chose for the delivery stream. Then choose Create delivery stream.