View a markdown version of this page

Amazon managed pipelines - Amazon SageMaker AI
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Amazon managed pipelines

An Amazon managed pipeline is a pipeline that Amazon creates and maintains for a common ML workflow. You can run it from your account without writing a pipeline definition. Amazon owns the pipeline, so its ARN uses aws in place of an account ID:

arn:aws:sagemaker:region:aws:pipeline/pipeline-name

When you run a managed pipeline, the execution runs in your account and uses an IAM role that you provide. The jobs that the execution creates, such as training jobs, and the data they write are in your account.

Find managed pipelines

To list managed pipelines, call ListPipelines with ManagedBy set to AWS. To list only the pipelines that you created, set it to CUSTOMER.

import boto3 sm = boto3.client("sagemaker") response = sm.list_pipelines(ManagedBy="AWS") for summary in response["PipelineSummaries"]: print(summary["PipelineArn"], summary.get("PipelineDescription"))

To see a managed pipeline's definition and parameters, call DescribePipeline with its ARN. To see its versions and their aliases, call ListPipelineVersions.

pipeline_arn = "arn:aws:sagemaker:us-west-2:aws:pipeline/pipeline-name" pipeline = sm.describe_pipeline(PipelineName=pipeline_arn) versions = sm.list_pipeline_versions(PipelineName=pipeline_arn)

Managed pipelines aren't available in every Amazon Web Services Region. If a managed pipeline isn't available to your account, requests for it fail with a ResourceNotFound error.

Run a managed pipeline

Call StartPipelineExecution with the pipeline ARN and a RoleArn. The execution uses this role to create and access resources in your account, so the role needs the permissions that the pipeline's steps require. RoleArn is required for a managed pipeline. You can't specify it for a pipeline that you created, because those executions use the pipeline's role.

response = sm.start_pipeline_execution( PipelineName=pipeline_arn, RoleArn="arn:aws:iam::111122223333:role/MySageMakerExecutionRole", PipelineParameters=[ {"Name": "ParameterName", "Value": "value"}, ], ) execution_arn = response["PipelineExecutionArn"]

The execution ARN contains your account ID. You monitor the execution the same way as an execution of your own pipeline, with DescribePipelineExecution and ListPipelineExecutionSteps. DescribePipelineExecution also returns the RoleArn that the execution uses.

Amazon can release new versions of a managed pipeline. If you don't specify a version, the execution uses the latest version that's available to you. To get the same behavior on every run, specify PipelineVersionId or PipelineVersionAlias. For more information about aliases, see Pipeline version aliases.

Customize a managed pipeline

You can't update or delete a managed pipeline, or change the aliases of its versions. These requests fail with an AccessDenied error. To change a managed pipeline, copy its definition into a pipeline in your account and edit the copy. Your copy doesn't receive the updates that Amazon makes to the managed pipeline.

definition = sm.describe_pipeline(PipelineName=pipeline_arn)["PipelineDefinition"] sm.create_pipeline( PipelineName="my-pipeline", PipelineDefinition=definition, RoleArn="arn:aws:iam::111122223333:role/MySageMakerExecutionRole", )