Concurrently iterate over items in a Step Functions workflow
In the previous topic, Define parallel tasks, you learned how to run separate branches of steps in parallel using the Parallel
state. Using the Map
state, you can run a set of workflow steps for each item in a dataset. The Map
state's iterations run in parallel, which makes it possible to process a dataset quickly.
By including the Map
state in your workflows you can perform tasks, such as data processing, using one of the two Map state processing modes: Inline mode and Distributed mode. To configure a Map
state, you define an ItemProcessor
, which contains JSON objects that specify the Map
state processing mode and its definition. You will run the Map
state in the default Inline mode, which supports up to 40 concurrent iterations. When you run the Map
state in Distributed mode, it supports up to 10,000 parallel child workflow executions.
When your workflow execution enters the Map
state, it will iterate over a JSON array specified in the state input. For each array item, its corresponding iteration runs in the context of the workflow that contains the Map
state. When all iterations are complete, the Map
state will return an array containing the output for each item processed by the ItemProcessor
.
You will use the Map
state in Inline mode to fetch the credit score of an applicant by iterating over a set of credit bureaus. To do this, you first fetch the names of all the credit bureaus stored in a Amazon DynamoDB table, and then use the Map
state to loop through the credit bureau list to fetch the applicant’s credit score reported by each of these bureaus.
Step 1: Create a DynamoDB table to store the name of all credit bureaus
In this step, you create a table named GetCreditBureau
using the DynamoDB console. The table uses the string attribute Name as the Partition key. In this table, you store the name of all the credit bureaus from which you want to fetch the applicant’s credit score.
Sign in to the Amazon Web Services Management Console and open the DynamoDB console at https://console.aws.amazon.com/dynamodb/.
In the navigation pane on the console, choose Tables, and then choose Create table.
Enter the table details as follows:
For the Table name, enter
GetCreditBureau
.For the Partition key, enter
Name
.Keep the default selections, and choose Create table.
After your table is created, in the Tables list, choose the GetCreditBureau table.
Choose Actions, and then choose Create item.
For Value, enter the name of a credit bureau. For example,
CredTrack
.Choose Create item.
Repeat this process and create items for names of other credit bureaus. For example,
KapFinn
andCapTrust
.
Step 2: Update the state machine – Fetch results from the DynamoDB table
In the Step Functions console, you’ll add a Task
state and use the Amazon SDK integration to fetch the names of credit bureaus from the DynamoDB table you created in Step 1. You’ll use the output of this step as the input for the Map
state you’ll add later in your workflow.
Open the CreditCardWorkflow state machine to update it.
Choose the Get list of credit bureaus state.
For API Parameters, specify the Table name value as
GetCreditBureau
.
Step 3: Create a Lambda function that returns the credit scores for all credit bureaus
In this step, you create a Lambda function that receives the names of all credit bureaus as input, and returns the credit score of the applicant for each of these credit bureaus. This Lambda function will be invoked from the Map
state you’ll add in your workflow.
Create a Node.js 16.x Lambda function and name it
get-credit-score
.On the page titled get-credit-score, paste the following code into the Code source area.
function getScore(arr) { let temp; let i = Math.floor((Math.random() * arr.length)); temp = arr[i]; console.log(i); console.log(temp); return temp; } const arrScores = [700, 820, 640, 460, 726, 850, 694, 721, 556]; exports.handler = (event, context, callback) => { let creditScore = getScore(arrScores); callback(null, "Credit score pulled is: " + creditScore + "."); };
Deploy the Lambda function.
Step 4: Update the state machine – add a Map state to iteratively fetch credit scores
In the Step Functions console, you add a Map
state that invokes the get-credit-score Lambda function to check the applicant’s credit score for all the credit bureaus returned by the Get list of credit bureaus state.
Open the CreditCardWorkflow state machine to update it.
Choose the Get scores from all credit bureaus state.
In the Configuration tab, choose Provide a path to items array and then enter
$.Items
.Choose Get all scores step inside the
Map
state.In the Configuration tab, make sure for Integration type, Optimized is selected.
For Function name, start typing the name of the get-credit-score Lambda function and choose it from the dropdown list that appears.
For Payload, choose No payload.
Next steps
In the next step, Run your workflow , you’ll learn how to run your workflow.