Using a Lambda function to continue a new execution - Amazon Step Functions
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China.

Using a Lambda function to continue a new execution

You can create a state machine that uses a Lambda function to start a new execution before the current execution terminates. Using this approach to continue your ongoing work in a new execution enables you to have a state machine that can break large jobs into smaller workflows, or to have a state machine that runs indefinitely.

This tutorial builds on the concept of using an external Lambda function to modify your workflow, which was demonstrated in the Iterating a Loop Using Lambda tutorial. You use the same Lambda function (Iterator) to iterate a loop for a specific number of times. In addition, you create another Lambda function to start a new execution of your workflow, and to decrement a count each time it starts a new execution. By setting the number of executions in the input, this state machine ends and restarts an execution a specified number of times.

The state machine you'll create implements the following states.

State Purpose

ConfigureCount

A Pass state that configures the count, index, and step values that the Iterator Lambda function uses to step through iterations of work.

Iterator

A Task state that references the Iterator Lambda function.

IsCountReached

A Choice state that uses a Boolean value from the Iterator function to decide whether the state machine should continue the example work, or move to the ShouldRestart state.

ExampleWork

A Pass state that represents the Task state that would perform work in an actual implementation.

ShouldRestart

A Choice state that uses the executionCount value to decide whether it should end one execution and start another, or simply end.

Restart

A Task state that uses a Lambda function to start a new execution of your state machine. Like the Iterator function, this function also decrements a count. The Restart state passes the decremented value of the count to the input of the new execution.

Prerequisites

Before you begin, go through the Creating a Step Functions state machine that uses Lambda tutorial to ensure that you're familiar with using Lambda and Step Functions together.

Step 1: Create a Lambda function to iterate a count

Note

If you have completed the Iterating a Loop Using Lambda tutorial, you can skip this step and use that Lambda function.

This section and the Iterating a Loop Using Lambda tutorial show how you can use a Lambda function to track a count, for example, the number of iterations of a loop in your state machine.

The following Lambda function receives input values for count, index, and step. It returns these values with an updated index and a Boolean named continue. The Lambda function sets continue to true if the index is less than count.

Your state machine then implements a Choice state that executes some application logic if continue is true, or moves on to ShouldRestart if continue is false.

Create the Iterate Lambda function

  1. Open the Lambda console, and then choose Create function.

  2. On the Create function page, choose Author from scratch.

  3. In the Basic information section, configure your Lambda function, as follows:

    1. For Function name, enter Iterator.

    2. For Runtime, choose Node.js 16.x.

    3. Keep all the default selections on the page, and then choose Create function.

      When your Lambda function is created, make a note of its Amazon Resource Name (ARN) in the upper-right corner of the page, for example:

      arn:aws-cn:lambda:us-east-1:123456789012:function:Iterator
  4. Copy the following code for the Lambda function into the Code source section of the Iterator page in the Lambda console.

    exports.handler = function iterator (event, context, callback) { let index = event.iterator.index; let step = event.iterator.step; let count = event.iterator.count; index = index + step; callback(null, { index, step, count, continue: index < count }) }

    This code accepts input values for count, index, and step. It increments the index by the value of step and returns these values, and the Boolean value of continue. The value of continue is true if index is less than count.

  5. Choose Deploy to deploy the code.

Test the Iterate Lambda function

To see your Iterate function working, run it with numeric values. You can provide input values for your Lambda function that mimic an iteration to see what output you get with specific input values.

To test your Lambda function

  1. In the Configure test event dialog box, choose Create new test event, and then type TestIterator for Event name.

  2. Replace the example data with the following.

    { "Comment": "Test my Iterator function", "iterator": { "count": 10, "index": 5, "step": 1 } }

    These values mimic what would come from your state machine during an iteration. The Lambda function increments the index and returns continue as true. When the index is not less than the count, it returns continue as false. For this test, the index has already incremented to 5. The results should increment the index to 6 and set continue to true.

  3. Choose Create.

  4. On the Iterator page in your Lambda console, be sure TestIterator is listed, and then choose Test.

    The results of the test are displayed at the top of the page. Choose Details and review the result.

    { "index": 6, "step": 1, "count": 10, "continue": true }
    Note

    If you set index to 9 for this test, the index increments to 10, and continue is false.

Step 2: Create a Restart Lambda function to start a new Step Functions execution

  1. Open the Lambda console, and then choose Create function.

  2. On the Create function page, choose Author from scratch.

  3. In the Basic information section, configure your Lambda function, as follows:

    1. For Function name, enter Restart.

    2. For Runtime, choose Node.js 16.x.

  4. Keep all the default selections on the page, and then choose Create function.

    When your Lambda function is created, make a note of its Amazon Resource Name (ARN) in the upper-right corner of the page, for example:

    arn:aws-cn:lambda:us-east-1:123456789012:function:Iterator
  5. Copy the following code for the Lambda function into the Code source section of the Restart page in the Lambda console.

    The following code decrements a count of the number of executions, and starts a new execution of your state machine, including the decremented value.

    var aws = require('aws-sdk'); var sfn = new aws.StepFunctions(); exports.restart = function(event, context, callback) { let StateMachineArn = event.restart.StateMachineArn; event.restart.executionCount -= 1; event = JSON.stringify(event); let params = { input: event, stateMachineArn: StateMachineArn }; sfn.startExecution(params, function(err, data) { if (err) callback(err); else callback(null,event); }); }
  6. Choose Deploy to deploy the code.

Step 3: Create a state machine

Now that you've created your two Lambda functions, create a state machine. In this state machine, the ShouldRestart and Restart states are how you break your work across multiple executions.

Example ShouldRestart Choice state

This excerpt of your state machine shows the ShouldRestartChoice state. This state determines whether you should restart the execution.

"ShouldRestart": { "Type": "Choice", "Choices": [ { "Variable": "$.restart.executionCount", "NumericGreaterThan": 1, "Next": "Restart" } ],

The $.restart.executionCount value is included in the input of the initial execution. It's decremented by one each time the Restart function is called, and then placed into the input for each subsequent execution.

Example Restart Task state

This excerpt of your state machine shows the RestartTask state. This state uses the Lambda function you created earlier to restart the execution, and to decrement the count to track the remaining number of executions to start.

"Restart": { "Type": "Task", "Resource": "arn:aws-cn:lambda:us-east-1:123456789012:function:Restart", "Next": "Done" },
  1. On the Step Functions console, choose Create state machine.

    Important

    Ensure that your state machine is under the same Amazon account and Region as the Lambda function you created earlier.

  2. Paste the following code into the Definition pane.

    { "Comment": "Continue-as-new State Machine Example", "StartAt": "ConfigureCount", "States": { "ConfigureCount": { "Type": "Pass", "Result": { "count": 100, "index": -1, "step": 1 }, "ResultPath": "$.iterator", "Next": "Iterator" }, "Iterator": { "Type": "Task", "Resource": "arn:aws-cn:lambda:us-east-1:123456789012:function:Iterator", "ResultPath": "$.iterator", "Next": "IsCountReached" }, "IsCountReached": { "Type": "Choice", "Choices": [ { "Variable": "$.iterator.continue", "BooleanEquals": true, "Next": "ExampleWork" } ], "Default": "ShouldRestart" }, "ExampleWork": { "Comment": "Your application logic, to run a specific number of times", "Type": "Pass", "Result": { "success": true }, "ResultPath": "$.result", "Next": "Iterator" }, "ShouldRestart": { "Type": "Choice", "Choices": [ { "Variable": "$.restart.executionCount", "NumericGreaterThan": 0, "Next": "Restart" } ], "Default": "Done" }, "Restart": { "Type": "Task", "Resource": "arn:aws-cn:lambda:us-east-1:123456789012:function:Restart", "Next": "Done" }, "Done": { "Type": "Pass", "End": true } } }
  3. Update the Resource string in the Restart and Iterator states to reference the respective Lambda functions you created earlier.

  4. Choose Next.

  5. On the Specify details page, enter a name for your state machine. For example, ContinueAsNew.

  6. Keep the default selections for all other options on the page, and then choose Create state machine.

  7. Save the Amazon Resource Name (ARN) of this state machine in a text file. You'll need to provide the ARN while providing permission to the Lambda function to start a new Step Functions execution.

Step 4: Update the IAM Policy

To ensure your Lambda function has permissions to start a new Step Functions execution, attach an inline policy to the IAM role you use for your Restart Lambda function. For more information, see Embedding Inline Policies in the IAM User Guide.

Note

You can update the Resource line in the previous example to reference the ARN of your ContinueAsNew state machine. This restricts the policy so that it can only start an execution of that specific state machine.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "states:StartExecution" ], "Resource": "arn:aws-cn:states:us-west-2:123456789012stateMachine:ContinueAsNew" } ] }

Step 5: Run an execution

To start an execution, provide input that includes the ARN of the state machine and an executionCount for how many times it should start a new execution.

  1. On the ContinueAsNew page, choose Start execution.

    The Start execution dialog box is displayed.

  2. (Optional) To identify your execution, you can specify a name for it in the Name box. By default, Step Functions generates a unique execution name automatically.

    Note

    Step Functions allows you to create state machine, execution, and activity names that contain non-ASCII characters. These non-ASCII names don't work with Amazon CloudWatch. To ensure that you can track CloudWatch metrics, choose a name that uses only ASCII characters.

  3. In the Input section, on the Start execution dialog box, enter the following as execution input:

    { "restart": { "StateMachineArn": "arn:aws-cn:states:us-east-1:123456789012:stateMachine:ContinueAsNew", "executionCount": 4 } }
  4. Update the StateMachineArn field with the ARN for your ContinueAsNew state machine.

  5. Choose Start Execution.

The Visual Workflow graph displays the first of the four executions. Before it completes, it will pass through the Restart state and start a new execution.


                First execution of four.

With this execution complete, you can look at the next execution that's running. Select the ContinueAsNew link at the top to see the list of executions. You should see both the recently closed execution, and an ongoing execution that the Restart Lambda function kicked off.


                One execution complete, the next one running.

When all the executions are complete, you should see four successful executions in the list. The first execution that was started displays the name you chose, and subsequent executions have a generated name.


                All executions complete.