Amazon SWF
Software workflows have been very popular and the preferred method to is break big tasks into sequential or parallel operational chunks depending on business requirements. Workflows mostly follow state machine concepts, where we have an orchestrator, execution steps in the form of states, and each state is executed through some events. Input/output and to/from states are managed by the orchestrator.
As defined by AWS, Amazon SWF helps developers to build, run, and scale background jobs that have sequential or parallel steps. It can be used in applications which need distributed asynchronous processing.
SWF web services help maintain the state of your workflow but they don't execute any software code, the logic of a workflow, or a state machine. SWF works on a polling mechanism, where your software source code has to poll for the tasks using SWF APIs. The software code will poll for new tasks and processes and provide its output back using SWF APIs.
Amazon SWF enables you to easily separate business logic from state management of your application that can be performed by human actions, web service calls, executable codes, and scripts.
SWF maintains the work tasks to be provided to your software code and keeps track of the workflow. SWF has two main parts: deciders and activity workers.
The decider application coordinates the execution of processing steps in a workflow. The flow of an activity task can be controlled by deciders. When change happens during the execution of a workflow, Amazon SWF will create a decision task. A decision task contains the workflow history and assign tasks to the decider. The decider will receive the decision task from Amazon SWF and analyze the workflow history to find out the next steps. Using decisions, the decider will communicate these steps back to Amazon SWF.
Activity workers are threads or processes to perform activity tasks which are part of the workflow. To use activity tasks, you must register them from the RegisterActivityType action or from the Amazon SWF console. The activity worker will poll Amazon SWF for a task which is appropriate for that worker to perform. Once the activity worker receives the task, it will process and complete it. Then it informs Amazon SWF of the result. After having completed the task, the activity worker will poll for another new task. It will continue until the workflow execution itself is complete. Activity workers can be running on a local machine, AWS EC2, or AWS Lambda.
As mentioned above, AWS SWF keeps track of the activity worker state. This tracking can be for activity workers taking a short time, or a long time (say for months) to finish its execution.
SWF is a reliable, scalable, and flexible solution.
AWS SWF can be used in following type of examples:
- Order processing system: Multiple tasks are required to be carried out in an order processing system. Each of these tasks can be programmed as an activity worker and decider applications can orchestrate the work between activity workers.
- Video and audio encoding/decoding: Large video or audio files can be broken into manageable chunks. These chunks can be encoded/decoded either in parallel or in sequential form. These processed chunks can be merged at the end of the process. In this encoding/decoding of each chunk, merging of all chunks can be looked as activity workers. The decider application will orchestrate which chunks are to be decoded/encoded in which sequence and when to merge processed chunks to a merged file.
AWS SWF provides the following methods for developers to communicate with it:
- AWS SWF APIs:
- Specify the names of workflows (it can also be achieved using the Management console).
- Start a new workflow. The workflow execution is kicked into action.
- From the activity worker machine, to request and execute tasks in the cloud or on-premises.
- AWS Flow Framework:
- Collection of libraries which assist developers to make it faster and easier to build applications with SWF.
- It creates and executes your application's steps, keeps track of its progress, and helps define retry rules for failed steps.
We have two sections about AWS SWF:
- The AWS SWF components section gives info about SWF components
- The AWS SWF examples section provides step-by-step information about SWF examples