Run a pipeline - Amazon SageMaker
Retrieve results from a pipeline run locallyRun a pipeline locally
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Run a pipeline

Start a new pipeline run with the pipeline.start() function as you would for a traditional SageMaker pipeline run. For information about the start() function, see sagemaker.workflow.pipeline.Pipeline.start.

Note

A step defined using the @step decorator runs as a training job. Therefore, be aware of the following limits:

  • Instance limits and training job limits in your accounts. Update your limits accordingly to avoid any throttling or resource limit issues.

  • The monetary costs associated with every run of a training step in the pipeline. For more details, refer to Amazon SageMaker Pricing.

Retrieve results from a pipeline run locally

To view the result of any step of a pipeline run, use execution.result(), as shown in the following snippet:

execution = pipeline.start()
execution.result(step_name="train")
Note

SageMaker Pipelines does not support execution.result() in local mode.

You can only retrieve results for one step at a time. If the step name was generated by SageMaker, you can retrieve the step name by calling list_steps as follows:

execution.list_step()

Run a pipeline locally

You can run a pipeline with @step-decorated steps locally as you would for traditional pipeline steps. For details about local mode pipeline runs, see Local Mode. To use local mode, provide a LocalPipelineSession instead of a SageMakerSession to your pipeline definition, as shown in the following example:

from sagemaker.workflow.function_step import step
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.pipeline_context import LocalPipelineSession

@step
def train():
    training_data = s3.download(....)
    ...
    return trained_model
    
step_train_result = train()

local_pipeline_session = LocalPipelineSession()

local_pipeline = Pipeline(
    name="<pipeline-name>",
    steps=[step_train_result],
    sagemaker_session=local_pipeline_session # needed for local mode
)

local_pipeline.create(role_arn="role_arn")

# pipeline runs locally
execution = local_pipeline.start()