Using Amazon Data Firehose with Amazon PrivateLink
You can use an interface VPC endpoint (Amazon PrivateLink) to access Amazon Data Firehose from
your VPC without requiring an Internet Gateway or NAT Gateway. Interface VPC endpoints don't
require an internet gateway, NAT device, VPN connection, or Amazon Direct Connect connection. Interface
VPC endpoints are powered by Amazon PrivateLink, an Amazon technology that enables private
communication between Amazon services using an elastic network interface with private IPs in
your Amazon VPC. For more information, see Amazon Virtual Private Cloud
Using interface VPC endpoints (Amazon PrivateLink) for Firehose
To get started, create an interface VPC endpoint in order for your Amazon Data Firehose traffic from your Amazon VPC resources to start flowing through the interface VPC endpoint. When you create an endpoint, you can attach an endpoint policy to it that controls access to Amazon Data Firehose. For more about using policies to control access from a VPC endpoint to Amazon Data Firehose, see Controlling Access to Services with VPC Endpoints.
The following example shows how you can set up an Amazon Lambda function in a VPC and create a VPC endpoint to allow the function to communicate securely with the Amazon Data Firehose service. In this example, you use a policy that allows the Lambda function to list the Firehose streams in the current Region but not to describe any Firehose stream.
Create a VPC endpoint
Sign in to the Amazon Web Services Management Console and open the Amazon VPC console at https://console.amazonaws.cn/vpc/
. -
In the VPC Dashboard choose Endpoints.
-
Choose Create Endpoint.
-
In the list of service names, choose
com.amazonaws.
.your_region
.kinesis-firehose -
Choose the VPC and one or more subnets in which to create the endpoint.
-
Choose one or more security groups to associate with the endpoint.
-
For Policy, choose Custom and paste the following policy:
{ "Statement": [ { "Sid": "Allow-only-specific-PrivateAPIs", "Principal": "*", "Action": [ "firehose:ListDeliveryStreams" ], "Effect": "Allow", "Resource": [ "*" ] }, { "Sid": "Allow-only-specific-PrivateAPIs", "Principal": "*", "Action": [ "firehose:DescribeDeliveryStream" ], "Effect": "Deny", "Resource": [ "*" ] } ] }
-
Choose Create endpoint.
Create an IAM role to use with the Lambda function
Open the IAM console at https://console.amazonaws.cn/iam/
. -
In the left pane, chose Roles and then choose Create role.
-
Under Select type of trusted entity, leave the default selection Amazon service.
-
Under Choose the service that will use this role, choose Lambda.
-
Choose Next: Permissions.
-
In the list of policies, search for and add the two policies named
AmazonLambdaVPCAccessExecutionRole
andAmazonDataFirehoseReadOnlyAccess
.Important
This is an example. You might need stricter policies for your production environment.
-
Choose Next: Tags. You don't need to add tags for the purpose of this exercise. Choose Next: Review.
-
Enter a name for the role, then choose Create role.
Create a Lambda function inside the VPC
Open the Amazon Lambda console at https://console.amazonaws.cn/lambda/
. -
Choose Create function.
-
Choose Author from scratch.
-
Enter a name for the function, then set Runtime to
Python 3.9
or higher. -
Under Permissions, expand Choose or create an execution role.
-
In the Execution role list, choose Use an existing role.
-
In the Existing role list, choose the role you created above.
-
Choose Create function.
-
Under Function code, paste the following code.
import json import boto3 import os from botocore.exceptions import ClientError def lambda_handler(event, context): REGION = os.environ['AWS_REGION'] client = boto3.client( 'firehose', REGION ) print("Calling list_delivery_streams with ListDeliveryStreams allowed policy.") delivery_stream_request = client.list_delivery_streams() print("Successfully returned list_delivery_streams request %s." % ( delivery_stream_request )) describe_access_denied = False try: print("Calling describe_delivery_stream with DescribeDeliveryStream denied policy.") delivery_stream_info = client.describe_delivery_stream(DeliveryStreamName='test-describe-denied') except ClientError as e: error_code = e.response['Error']['Code'] print ("Caught %s." % (error_code)) if error_code == 'AccessDeniedException': describe_access_denied = True if not describe_access_denied: raise else: print("Access denied test succeeded.")
-
Under Basic settings, set the timeout to 1 minute.
-
Under Network, choose the VPC where you created the endpoint above, then choose the subnets and security group that you associated with the endpoint when you created it.
-
Near the top of the page, choose Save.
-
Choose Test.
-
Enter an event name, then choose Create.
-
Choose Test again. This causes the function to run. After the execution result appears, expand Details and compare the log output to the function code. Successful results show a list of the Firehose streams in the Region, as well as the following output:
Calling describe_delivery_stream.
AccessDeniedException
Access denied test succeeded.
Supported Amazon Web Services Regions
Interface VPC endpoints are currently supported within the following regions.
-
US East (Ohio)
-
US East (N. Virginia)
-
US West (N. California)
-
US West (Oregon)
-
Asia Pacific (Mumbai)
-
Asia Pacific (Seoul)
-
Asia Pacific (Singapore)
-
Asia Pacific (Sydney)
-
Asia Pacific (Tokyo)
-
Asia Pacific (Hong Kong)
-
Canada (Central)
-
Canada West (Calgary)
-
China (Beijing)
-
China (Ningxia)
-
Europe (Frankfurt)
-
Europe (Ireland)
-
Europe (London)
-
Europe (Paris)
-
South America (São Paulo)
-
Amazon GovCloud (US-East)
-
Amazon GovCloud (US-West)
-
Europe (Spain)
-
Middle East (UAE)
-
Asia Pacific (Jakarta)
-
Asia Pacific (Osaka)
-
Israel (Tel Aviv)
-
Asia Pacific (Malaysia)