Amazon Simple Workflow Service (SWF) Overview

Amazon Simple Workflow Service (SWF) helps developers build, run, and scale background jobs that have parallel or sequential steps. You can think of Amazon SWF as a fully-managed state tracker and task coordinator in the Cloud.

It makes easy to coordinate work across distributed application components. The tasks can be performed by executable code, web service calls, human actions and/or scripts. Supports range of use cases including media processing, business process work flows, and analytics pipeline (see use case summary).

Use Cases Summary (based on AWS doc)

1. Video Encoding Using S3 and EC2.

   1. Large videos are uploaded to S3 in chunks.

   2. After a chunk is uploaded, it is encoded by downloading it to an EC2 instance and the result is stored to another S3 location. After all of the chunks have been encoded, they are combined into a complete encoded file which is stored back in its entirety to S3.

   3. Failures such as encoding errors need to be detected and handled through SWF's cloud workflow management.

2. Data Center Migration (private datacenter to cloud)

   1. Business critical operations need to be moved entirely to the cloud without causing disruptions.

   2. SWF-based applications can combine workers that wrap components running in the datacenter with workers that run in the cloud.

   3. To transition a datacenter worker seamlessly, new workers of the same type are first deployed in the cloud and run along with the workers in the datacenter. Once cloud-based workers are tested and validated by routing a portion of the load through them, the workers in the datacenter are gradually stopped and those in the cloud are scaled up, so that they move entirely to a cloud workflow management application.

   4. This cloud workflow process can be repeated for all other workers in the datacenter so that the application moves entirely to the clou

3. Product Catalogs With Human Workers Using Amazon Mechanical Turk

   1. While validating data in large catalogs, the products in the catalog are processed in batches. Different batches can be processed concurrently.

   2. For each batch, the product data is extracted from servers in the datacenter and transformed into CSV files required by Amazon Mechanical Turk’s Requester User Interface (RUI). The CSV is uploaded to populate and run the HITs (Human Intelligence Tasks).

   3. When HITs complete, the resulting CSV file is reverse transformed to get the data back into the original format. The results are then assessed and Amazon Mechanical Turk workers are paid for acceptable results.

   4. Failures are weeded out and reprocessed, while the acceptable HIT results are used to update the catalog. Failed HITs are re-batched and sent through the pipeline again.

Benefits Summary (based on AWS docs)

1. Logical Separation

   1. SWF promotes a separation between the control flow (stepwise logic) and the actual business logic. This allows you to separately manage, maintain, and scale "state machinery" of application without affecting business logic and similarly change business logic without affecting the "state machinery". 

2. Reliable

   1. SWF runs within Amazon’s high-availability data centers, and redundantly stores the tasks, reliably dispatches them to application components, tracks their progress, and keeps their latest state.

3. Simple

   1. SWF replaces the complexity of custom-coded workflow solutions and process automation software with a fully managed cloud workflow web service.

4. Scalable

   1. SWF seamlessly scales with your application’s usage without any manual administration.

5. Flexible

   1. SWF lets you write your application components and coordination logic in any programming language and run them in the cloud or on-premises.