Discover use cases for Step Functions workflows
With Amazon Step Functions, you can build workflows that manage state over time, make decisions based on incoming data, and handle errors and exceptions.
Data processing
As the volume of data grows, coming from increasingly diverse sources, organizations find they need to move quickly to process this data to ensure they make faster, well-informed business decisions. To process data at scale, organizations need to elastically provision resources to manage the information they receive from mobile devices, applications, satellites, marketing and sales, operational data stores, infrastructure, and more.
Step Functions provides the scalability, reliability, and availability needed to successfully
manage your data processing workflows. You can manage millions of concurrent executions
with Step Functions as it scales horizontally and provides fault-tolerant workflows. Process data
faster using parallel executions like Step Functions’ Parallel workflow state state type, or dynamic
parallelism using its Map workflow state state type. As part of your
workflow, you can use the Map workflow state state to iterate over
objects in a static data store like an Amazon S3 bucket. Step Functions also lets you easily retry failed
executions, or choose a specific way to handle errors without the need to manage a complex
process.
Depending upon your data processing needs, Step Functions directly integrates with other data processing services provided by Amazon such as Amazon Batch for batch processing, Amazon EMR for big data processing, Amazon Glue for data preparation, Athena for data analysis, and Amazon Lambda for compute.
Examples of the types of data processing workflows that customers use Step Functions to accomplish include:
File, video, and image processing
-
Take a collection of video files and convert them to other sizes or resolutions that are ideal for the device they will be displayed on, such as mobile phones, laptops, or a television.
-
Take a large collection of photos uploaded by users and convert them into thumbnails or various resolution images that can then be displayed on users’ websites.
-
Take semi-structured data, such as a CSV file, and combine it with unstructured data, such as an invoice, to produce a business report that is sent to business stakeholders monthly.
-
Take earth observing data collected from satellites, convert it into formats that align with each other and then add other data sources collected on earth for additional insight.
-
Take the transportation logs from various modes of transportation for products and look for optimizations using Monte Carlo Simulations and then send reports back to the organizations and people that are relying on you to ship their goods.
Coordinate extract, transform and load (ETL) jobs:
-
Combine sales opportunity records with marketing metric datasets through a series of data preparation steps using Amazon Glue, and produce business intelligence reports that can be used across the organization.
-
Create, start, and terminate an Amazon EMR cluster for big data processing.
Batch processing and High Performance Computing (HPC) workloads:
-
Build a genomics secondary analysis pipeline that processes raw whole genome sequences into variant calls. Align raw files to a reference sequence, and call variants on a specified list of chromosomes using dynamic parallelism.
-
Find efficiencies in the production of your next mobile device or other electronics by simulating various layouts using different electric and chemical compounds. Run large batch processing of your workloads through various simulations to get the optimal design.
Machine learning
Machine learning enables organizations to quickly analyze collected data to identify patterns, then make decisions with minimal human intervention. Machine learning starts with an initial set of data, known as training data. This training data helps to increase a machine learning model’s prediction accuracy, and serves as the foundation through which this model learns. Once the model is considered accurate enough to meet business needs, it’s deployed to production. The Amazon Step Functions Data Science Software Development Kit (SDK) is an open-source library that allows you to easily create workflows that preprocess data, train and then publish your models using Amazon SageMaker and Step Functions.
Preprocessing existing data sets is how an organization often creates training data. This method adds information, such as by labeling objects in an image, annotating text or processing audio. To preprocess data you can use Amazon Glue, or you can create an SageMaker notebook instance that runs the Jupyter Notebook app. Once your data is ready, it can be uploaded to Amazon S3 for easy access. As machine learning models are trained, you can make adjustments to each model’s parameters to improve accuracy until it’s ready for deployment.
Step Functions lets you to orchestrate end-to-end machine learning workflows on SageMaker. These workflows can include data preprocessing, post-processing, feature engineering, data validation, and model evaluation. Once the model has been deployed to production, you can refine and test new approaches to continually improve business outcomes. You can create production-ready workflows directly in Python, or you can use the Step Functions Data Science SDK to copy that workflow, experiment with new options, and place the refined workflow in production.
Some types of machine learning workflows that customers use Step Functions for include:
Fraud Detection
-
Identify and prevent fraudulent transactions, such as credit fraud, from occurring.
-
Detect and prevent account takeovers using trained machine learning models.
-
Identify promotional abuse, including the creation of fake accounts, so you can quickly take action.
Personalization and Recommendations
-
Recommend products to targeted customers based upon what is predicted to attract their interest.
-
Predict whether a customer will upgrade their account from a free tier to a paid subscription.
Data Enrichment
-
Use data enrichment as part of preprocessing to provide better training data for more accurate machine learning models.
-
Annotate text and audio excerpts to add syntactical information, such as sarcasm and slang.
-
Label additional objects in images to provide critical information for the model to learn from, such as whether an object is an apple, a basketball, a rock, or an animal.
Microservice orchestration
Microservice architecture breaks applications into loosely coupled services. Benefits include improved scalability, increased resiliency, and faster time to market. Each microservice is independent, making it easy to scale up a single service or function without needing to scale the entire application. Individual services are loosely coupled, letting independent teams focus on a single business process, without the need for them to understand the entire application. Microservices also let you choose which individual components suit your business needs, giving you the flexibility to change your selection without rewriting your entire workflow. Different teams can use the programming languages and frameworks of their choice to work with their microservice, and this microservice can still communicate with any other in the application through application programming interfaces (APIs).
Step Functions gives you several ways to manage your microservice workflows. For long-running workflows you can use Standard Workflows with the Amazon Fargate integration to orchestrate applications running in containers. For short-duration, high-volume workflows that require an immediate response, Synchronous Express Workflows are ideal. These can be used for web-based or mobile applications, which often have workflows of short duration, and require the completion of a series of steps before they return a response. You can directly trigger a Synchronous Express Workflows from Amazon API Gateway, and the connection is held open until the workflow completes or timeouts. For short duration workflows that do not require an immediate response, Step Functions provides Asynchronous Express Workflows.
Examples of some API orchestrations that use Step Functions include:
Synchronous or real-time workflows
-
Change a value in a record such as updating an employee’s last name and have the change immediately visible on the screen.
-
Update an order during checkout, such as adding, removing, or changing the quantity of an item, then immediately reflect the update back to the customer.
-
Run a quick processing job and immediately return the result back to the requester.
Container Orchestration
-
Run jobs on Kubernetes with Amazon Elastic Kubernetes Service or on Amazon Elastic Container Service (ECS) with Fargate and integrate with other Amazon services, such as sending notifications with Amazon SNS, as part of the same workflow.
IT and security automation
IT automation can help manage increasingly complex and time-consuming operations, such as upgrading and patching software, deploying security updates to address vulnerabilities, selecting infrastructure, synchronizing data, routing support tickets, and more. The automation of repetitive and time-consuming tasks can allow your organization to complete routine operations quickly and consistently on a large scale. This lets you focus on strategic work such as feature development, complex support requests, and innovation while meeting these growing demands.
Step Functions allows you to create workflows that automatically scale to meet the needs of your business without requiring manual intervention. In cases where an error occurs in your workflow, it often does not require manual intervention. Step Functions lets you automatically retry failed tasks and an exponential backoff that can manage errors in your workflow.
There can be situations where human intervention is required before the workflow can progress. For example, approving a substantial credit increase may require human approval. To manage this, you can define branching logic in Step Functions, so that only requests over a defined amount require human approval, while all other requests are automatically completed. In cases where human approval is required, Step Functions lets you pause the workflow at a specific step, wait for a response, and then continue the workflow once the response is received.
Some examples of the types of automation workflows that customers use Step Functions for include:
IT automation
-
Auto-remediate incidents like opening an SSH port, low disk space, or when a public access is given to a Amazon S3 bucket.
-
Automate the deployment of Amazon CloudFormation StackSets
Security automation
-
Automate the response to a scenario where a user and user access key has been exposed.
-
Auto-remediate security incident responses according to policy actions defined such as restricting actions to specific ARNs or applying other actions.
-
Warn employees of phishing emails within seconds of receipt.
Human Approval
-
Automate the training of machine learning model and then require manual approval of the model by a data scientist before then automatically deploying or rejecting the model based upon the response received.
-
Automate the routing of customer feedback received based on sentiment analysis so that those with a negative sentiment are immediately escalated for manual review.