Setting up Neptune ML without using the quick-start Amazon CloudFormation template
1. Start with a working Neptune DB cluster
If you don't use the Amazon CloudFormation quick-start template to set up Neptune ML, you will need an existing Neptune DB cluster to work with. If you want, you can use one you already have, or clone one that you are already using, or you can create a new one (see Create a DB cluster).
Make sure that the Neptune DB cluster you will be using is running at least
engine version 1.0.5.0
. If it's running an earlier engine version, you
can upgrade it as described in Neptune engine updates.
2. Install the Neptune-Export service
If you haven't already done so, install the Neptune-Export service, as explained in Using the Neptune-Export service to export Neptune data.
Add an inbound rule to the NeptuneExportSecurityGroup
security group
that the install creates, with the following settings:
Type:
Custom TCP
Protocol:
TCP
Port range:
80 - 443
Source:
(Neptune DB cluster security group ID)
3. Create a custom NeptuneLoadFromS3
IAM role
If you have not already done so, create a custom NeptuneLoadFromS3
IAM
role, as explained in Creating an IAM role to access Amazon S3.
Create a custom NeptuneSageMakerIAMRole
role
Use the IAM consoleNeptuneSageMakerIAMRole
, using the following policy:
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "ec2:CreateNetworkInterface", "ec2:CreateNetworkInterfacePermission", "ec2:CreateVpcEndpoint", "ec2:DeleteNetworkInterface", "ec2:DeleteNetworkInterfacePermission", "ec2:DescribeDhcpOptions", "ec2:DescribeNetworkInterfaces", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcEndpoints", "ec2:DescribeVpcs" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "ecr:GetAuthorizationToken", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "iam:PassRole" ], "Resource": [ "arn:aws:iam::*:role/*" ], "Condition": { "StringEquals": { "iam:PassedToService": [ "sagemaker.amazonaws.com" ] } }, "Effect": "Allow" }, { "Action": [ "kms:CreateGrant", "kms:Decrypt", "kms:GenerateDataKey*" ], "Resource": "arn:aws:kms:*:*:key/*", "Effect": "Allow" }, { "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents" ], "Resource": [ "arn:aws:logs:*:*:log-group:/aws/sagemaker/*" ], "Effect": "Allow" }, { "Action": [ "sagemaker:CreateEndpoint", "sagemaker:CreateEndpointConfig", "sagemaker:CreateHyperParameterTuningJob", "sagemaker:CreateModel", "sagemaker:CreateProcessingJob", "sagemaker:CreateTrainingJob", "sagemaker:CreateTransformJob", "sagemaker:DeleteEndpoint", "sagemaker:DeleteEndpointConfig", "sagemaker:DeleteModel", "sagemaker:DescribeEndpoint", "sagemaker:DescribeEndpointConfig", "sagemaker:DescribeHyperParameterTuningJob", "sagemaker:DescribeModel", "sagemaker:DescribeProcessingJob", "sagemaker:DescribeTrainingJob", "sagemaker:DescribeTransformJob", "sagemaker:InvokeEndpoint", "sagemaker:ListTags", "sagemaker:ListTrainingJobsForHyperParameterTuningJob", "sagemaker:StopHyperParameterTuningJob", "sagemaker:StopProcessingJob", "sagemaker:StopTrainingJob", "sagemaker:StopTransformJob", "sagemaker:UpdateEndpoint", "sagemaker:UpdateEndpointWeightsAndCapacities" ], "Resource": [ "arn:aws:sagemaker:*:*:*" ], "Effect": "Allow" }, { "Action": [ "sagemaker:ListEndpointConfigs", "sagemaker:ListEndpoints", "sagemaker:ListHyperParameterTuningJobs", "sagemaker:ListModels", "sagemaker:ListProcessingJobs", "sagemaker:ListTrainingJobs", "sagemaker:ListTransformJobs" ], "Resource": "*", "Effect": "Allow" }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::*" ], "Effect": "Allow" } ] }
While creating this role, edit the trust relationship so that it reads as follows:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com", "rds.amazonaws.com", "sagemaker.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }
Finally, copy the ARN assigned to this new NeptuneSageMakerIAMRole
role.
Configure your DB cluster to enable Neptune ML
To set up your DB cluster for Neptune ML
-
In the Neptune console
, navigate to Parameter Groups and then to the DB cluster parameter group associated with the DB cluster you will be using. Set the neptune_ml_iam_role
parameter to the ARN assigned to theNeptuneSageMakerIAMRole
role that you just created. -
Navigate to Databases, then select the DB cluster you will be using for Neptune ML. Select Actions then Manage IAM roles.
On the Manage IAM roles page, select Add role and add the
NeptuneSageMakerIAMRole
. Then add theNeptuneLoadFromS3
role.Reboot the writer instance of your DB cluster.
Create two SageMaker endpoints in your Neptune VPC
Finally, to give the Neptune engine access the necessary SageMaker management APIs, you need to create two SageMaker endpoints in your Neptune VPC, as explained in Create two endpoints for SageMaker in your Neptune VPC.