Using Amazon CloudFormation to Set Up Neptune-to-Neptune Replication with the Streams Consumer Application
You can use an Amazon CloudFormation template to set up the Neptune streams consumer application to support Neptune-to-Neptune replication.
Topics
Choose an Amazon CloudFormation template for Your Region
To launch the appropriate Amazon CloudFormation stack on the Amazon CloudFormation console, choose one of the Launch Stack buttons in the following table, depending on the Amazon Region that you want to use.
Region | View | View in Designer | Launch |
---|---|---|---|
US East (N. Virginia) | View |
View in Designer |
|
US East (Ohio) | View |
View in Designer |
|
US West (N. California) | View |
View in Designer |
|
US West (Oregon) | View |
View in Designer |
|
Canada (Central) | View |
View in Designer |
|
South America (São Paulo) | View |
View in Designer |
|
Europe (Stockholm) | View |
View in Designer |
|
Europe (Ireland) | View |
View in Designer |
|
Europe (London) | View |
View in Designer |
|
Europe (Paris) | View |
View in Designer |
|
Europe (Frankfurt) | View |
View in Designer |
|
Middle East (Bahrain) | View |
View in Designer |
|
Middle East (UAE) | View |
View in Designer |
|
Israel (Tel Aviv) | View |
View in Designer |
|
Africa (Cape Town) | View |
View in Designer |
|
Asia Pacific (Tokyo) | View |
View in Designer |
|
Asia Pacific (Hong Kong) | View |
View in Designer |
|
Asia Pacific (Seoul) | View |
View in Designer |
|
Asia Pacific (Singapore) | View |
View in Designer |
|
Asia Pacific (Sydney) | View |
View in Designer |
|
Asia Pacific (Mumbai) | View |
View in Designer |
|
China (Beijing) | View |
View in Designer |
|
China (Ningxia) | View |
View in Designer |
|
Amazon GovCloud (US-West) | View |
View in Designer |
|
Amazon GovCloud (US-East) | View |
View in Designer |
|
On the Create Stack page, choose Next.
Add details About the Neptune streams consumer stack you're creating
The Specify Stack Details page provides properties and parameters that you can use to control the setup of the application:
Stack Name –
The name of the new Amazon CloudFormation stack that you're creating. You can generally use
the default value, NeptuneStreamPoller
.
Under Parameters, provide the following:
Network configuration for the VPC Where the streams consumer runs
VPC
– Provide the name of the VPC where the polling Lambda function will run.SubnetIDs
– The subnets to which a network interface is established. Add subnets corresponding to your Neptune cluster.SecurityGroupIds
– Provide the IDs of security groups that grant write inbound access to your source Neptune DB cluster.RouteTableIds
– This is needed to create an Amazon DynamoDB endpoint in your Neptune VPC, if you do not already have one. You must provide a comma-separated list of route table IDs associated with the subnets.CreateDDBVPCEndPoint
– A Boolean value that defaults totrue
, indicating whether or not it is necessary to create a Dynamo DB VPC endpoint. You only need to change it tofalse
if you have already created a DynamoDB endpoint in your VPC.CreateMonitoringEndPoint
– A Boolean value that defaults totrue
, indicating whether or not it is necessary to create a monitoring VPC endpoint.. You only need to change it tofalse
if you have already created a monitoring endpoint in your VPC.
Stream Poller
ApplicationName
– You can generally leave this set to the default (NeptuneStream
). If you use a different name, it must be unique.LambdaMemorySize
– Used to set the memory size available to the Lambda poller function. The default value is 2,048 megabytes.LambdaRuntime
– The language used in the Lambda function that retrieves items from the Neptune stream. You can set this either topython3.9
or tojava8
.LambdaS3Bucket
– The Amazon S3 bucket that contains Lambda code artifacts. Leave this blank unless you are using a custom Lambda polling function that loads from a different Amazon S3 bucket.LambdaS3Key
– The Amazon S3 key that corresponds to your Lambda code artifacts. Leave this blank unless you are using a custom Lambda polling function.LambdaLoggingLevel
– In general, leave this set to the default value, which isINFO
.ManagedPolicies
– Lists the managed policies to use for execution of your Lambda function. In general, leave this blank unless you are using a custom Lambda polling function.StreamRecordsHandler
– In general, leave this blank unless you are using a custom handler for the records in Neptune streams.StreamRecordsBatchSize
– The maximum number of records to be fetched from stream. You can use this parameter to tune performance. The default (5000
) is a good place to start. The maximum allowable is 10,000. The higher the number, the fewer network calls are needed to read records from the stream, but the more memory is required to process the records. Lower values of this parameter result in lower throughput.MaxPollingWaitTime
– The maximum wait time between two polls (in seconds). Determines how frequently the Lambda poller is invoked to poll the Neptune streams. Set this value to 0 for continuous polling. The maximum value is 3,600 seconds (1 hour). The default value (60 seconds) is a good place to start, depending on how fast your graph data changes.MaxPollingInterval
– The maximum continuous polling period (in seconds). Use this to set a timeout for the Lambda polling function. The value should be in the range between 5 seconds and 900 seconds. The default value (600 seconds) is a good place to start.StepFunctionFallbackPeriod
– The number of units of step-function-fallback-period to wait for the poller, after which the step function is called through Amazon CloudWatch Events to recover from a failure. The default (5 minutes) is a good place to start.StepFunctionFallbackPeriodUnit
– The time units used to measure the precedingStepFunctionFallbackPeriodUnit
(minutes
,hours
, ordays
). The default (minutes
) is generally sufficient.
Neptune stream
-
NeptuneStreamEndpoint
– (Required) The endpoint of the Neptune source stream. This takes one of two forms:https://
(or its alias,your DB cluster
:port
/propertygraph/streamhttps://
).your DB cluster
:port
/pg/streamhttps://
.your DB cluster
:port
/sparql/stream
Neptune Query Engine
– Choose Gremlin, openCypher, or SPARQL.IAMAuthEnabledOnSourceStream
– If your Neptune DB cluster is using IAM authentication, set this parameter totrue
.StreamDBClusterResourceId
– If your Neptune DB cluster is using IAM authentication, set this parameter to the cluster resource ID. The resource ID is not the same as the cluster ID. Instead, it takes the form:cluster-
followed by 28 alpha-numeric characters. It can be found under Cluster Details in the Neptune console.
Target Neptune DB cluster
-
TargetNeptuneClusterEndpoint
– The cluster endpoint (hostname only) of the target backup cluster.Note that if you specify
TargetNeptuneClusterEndpoint
, you cannot also specifyTargetSPARQLUpdateEndpoint
. -
TargetNeptuneClusterPort
– The port number for the target cluster.Note that if you specify
TargetSPARQLUpdateEndpoint
, the setting forTargetNeptuneClusterPort
is ignored. -
IAMAuthEnabledOnTargetCluster
– Set to true if IAM authentication is to be enabled on the target cluster. -
TargetAWSRegion
– The target backup cluster's Amazon region, such asus-east-1
). You must provide this parameter only when the Amazon region of the target backup cluster is different from the region of the Neptune source cluster, as in the case of cross-region replication. If the source and target regions are the same, this parameter is optional.Note that if the
TargetAWSRegion
value is not a valid Amazon region that Neptune supports, the process fails. -
TargetNeptuneDBClusterResourceId
– Optional: this is only needed when IAM authentication is enabled on the target DB cluster. Set to the resource ID of the target cluster. -
SPARQLTripleOnlyMode
– Boolean flag that determines whether triple-only mode is enabled. In triple-only mode, there is no named-graph replication. The default value isfalse
. -
TargetSPARQLUpdateEndpoint
– URL of the target endpoint for SPARQL update, such ashttps://abc.com/xyz
. This endpoint can be any SPARQL store that supports quad or triples.Note that if you specify
TargetSPARQLUpdateEndpoint
, you cannot also specifyTargetNeptuneClusterEndpoint
, and the setting ofTargetNeptuneClusterPort
is ignored. -
BlockSparqlReplicationOnBlankNode
– Boolean flag which, if set totrue
, stops replication for BlankNode in SPARQL (RDF) data. The default value isfalse
.
Alarm
Required to create Cloud watch Alarm
– Set this totrue
if you want to create a CloudWatch alarm for the new stack.SNS Topic ARN for Cloudwatch Alarm Notifications
– The SNS topic ARN where CloudWatch alarm notifications should be sent (only needed if alarms are enabled).Email for Alarm Notifications
– The email address to which alarm notifications should be sent (only needed if alarms are enabled).
For destination of the alarm notification, you can add SNS only, email only, or both SNS and email.
Run the Amazon CloudFormation Template
Now you can complete the process of provisioning a Neptune streams consumer application instance as follows:
In Amazon CloudFormation, on the Specify Stack Details page, choose Next.
On the Options page, choose Next.
-
On the Review page, select the first check box to acknowledge that Amazon CloudFormation will create IAM resources. Select the second check box to acknowledge
CAPABILITY_AUTO_EXPAND
for the new stack.Note
CAPABILITY_AUTO_EXPAND
explicitly acknowledges that macros will be expanded when creating the stack, without prior review. Users often create a change set from a processed template so that the changes made by macros can be reviewed before actually creating the stack. For more information, see the Amazon CloudFormation CreateStack API in the Amazon CloudFormation API Reference.Then choose Create.
To update the stream poller with the latest Lambda artifacts
You can update the stream poller with the latest Lambda code artifacts as follows:
In the Amazon Web Services Management Console, navigate to Amazon CloudFormation and select the main parent Amazon CloudFormation stack.
Select the Update option for the stack.
Select Replace current template.
-
For the template source, choose Amazon S3 URL and enter the following S3 URL:
https://aws-neptune-customer-samples.s3.amazonaws.com/neptune-stream/neptune_to_neptune.json
Select Next without changing any Amazon CloudFormation parameters.
Choose Update Stack.
The stack will now update the Lambda artifacts with the most recent ones.