MicroWorkflow 1.3.4

MicroWorkflow .net

0.

1. Design goals

Simplicity

• We model only the steps in a workflow, not the transitions between them
  • This greatly simplify the model, the versioning of flows and steps
  • enable you to use reusable code blocks for determining a transition
• It is easy to embed it directly into your solutions to improve resiliance or use as a stand-alone workflow

We use C# all the way

• We don't want to invent a new language - we love C#!
• Workflow code is readable, debugable, and testable - like the rest of your code base
• You can use existing best practices for logging, IOC containers etc.
• Workflow code is easy to commit and merge use your existing branching strategies
• You do not need a special graphical editor for specifying flows

The datamodel is simple - just three DB tables

• If things break in production, it is easy for you to figure out what has happened and remedy the problem
• You can reason about the consequences of versioning the step implementations vs. simply change the existing flows

Scalable

• You can add more workers in the workflow engine (vertical scaling)
• You can add more servers each running a workflow engine (horizontal scaling)

No external dependencies

• The core library has no external dependencies, you can use whatever database, logger, json/xml/binary serializer you want ... in any version you want
• Convenience supplement nuget packages for Newtonsoft json, Ado .net, and Autofac are provided

2. Overview

Supported scalabilities

3. Getting started

To define a workflow with the two steps FetchData (which fetches some data), and AnalyzeWords (that analyzes the data), we implement interface IStepImplementation twice. To transition from one step to one (or several steps), use Done(). This tells the engine that the current step has finished sucesfully. You can supply one or more steps that will be executed in the future. This is how you control ordering of events.

There are no restrictions on the names of steps, but we found using a scheme similar to REST api's is beneficial. Hence we recommend you to use {version}/{business domain}/{workflow name}/{workflow step}.

[StepName(Name)]
class FetchData : IStepImplementation
{
    public const string Name = "v1/demos/fetch-word-analyze-email/fetch";

    public async Task<ExecutionResult> ExecuteAsync(Step step)
    {
        ... 
        return step.Done()
                .With(new Step(AnalyzeWords.Name, state-for-step));
    }
}

[StepName(Name)]
class AnalyzeWords : IStepImplementation
{
    public const string Name = "v1/demos/fetch-wordanalyze-email/process";

    public async Task<ExecutionResult> ExecuteAsync(Step step)
    {
        ...        
        return step.Done();
    }
}

Below are some more elaborate exaples.

Simple console demo

A fully working C# example in one file: https://github.com/kbilsted/MicroWorkflow.net/blob/master/src/Demos/ConsoleDemo/Program.cs

Webapi demo

Now that you have understood the basics, lets make an example using a real IOC container and a database see https://github.com/kbilsted/MicroWorkflow.net/tree/master/src/Demos/WebApiDemo

IOC container

You can use any IOC container that supports named instances. We use Autofac. For more information see https://github.com/kbilsted/MicroWorkflow.net/tree/master/src/Product/MicroWorkflow.Ioc.Autofac

Database

You likely want to persist workflows in a database. We currently use Microsoft SQL Server in production environments, but and SQL database should be easy to get working. For more information see https://github.com/kbilsted/MicroWorkflow.net/tree/master/src/Product/MicroWorkflow.AdoPersistence

4. Core concepts in Micro Workflow

The model revolves around the notion of a step. A step is in traditional workfow litterature referred to as an activity. Where activities live in a workflow. The workflow has identity and state and so forth. In Micro Workflow, however, there is only a FlowId property. No modelling of transitions nor workflow state. It is often desireable to store state around your business entities, in fact it is highly encouraged that you keep doing this.

A step has the following properties

• A step has a name that identifies what code to execute when the steps executes
• A step has a state which can be deserialed during execution and is serialized after execution. It is not uncommon that it simply holds a reference to data in other database tables.
• A schedule date that denote the earliest execution time of the step.
• A singleton flag denoting that only a single step with that name can exist in the ready queue. This is useful for monitoring steps or scenarios where multiple servers are in use.
• A step lives in either of 3 queues:
  • ready (execution has not yet started, or the step has executed with errors and is automatically retried).
  • failed (execution is given up and not retried),
  • done (succesful execution).
• During a step execution a step can spawn one or many new steps. Hence forming a chain or a graph of things to do. These steps execute after the current step.
• Each step has a number of tracking fields such as create date, execution time, correlation id, flow id, created by id.
• There are a few more fields, they are all documented here https://github.com/kbilsted/MicroWorkflow.net/blob/master/src/Product/MicroWorkflow/Step.cs

Orthogonal to the step data we have step implementations.

• These are code blocks with names.
• An implementation may have multiple names, this is very useful in a number of situations
  • When reusing step implementations across versions or across multiple workflows
  • when refactoring step names
  • It provides better loggin context information, for example, rather than using a generic name like "send email", we can use a descriptive name "v1/customer-onboarding/send-email" and "v1/customer-churn/send-email", making it easy to identify which flows and business impact on failure..

Operations you can do on steps

• A ready step may be activated meaning that it changes scheduling time, and activation parameters may be given.
• A done/failed step may be re-executed, by making a copy of it and adding it to the ready queue.

5. Performance

Simplicify is the focus of the code base. Performance is simply a side-effect of keeping things simple.

On a 2020 mid-tier computer we execute 10.000/sec steps using a single Workflow engine with 8 workers and an un-optimized SQL Server instance.

Your milage may wary, so I highly recommend you do your own measurements before jumping to any conclusions. You can take outset in some simple test scenarios at https://github.com/kbilsted/MicroWorkflow.net/blob/master/src/Demos/MicroWorkFlow.Tests/PerformanceTests.cs

6. Flow versioning

Since each step may be regarded as being part of a flow, or as a single independent step, there is no notion of versions. However, you can use a version number in steps (similar to using version in REST api's). This enable you to create a new version with new steps that has a different implementation to the old. This way existing steps can operate on the old code, and new steps operate on the new code. If all steps need to execute on the new code, simply use multiple step names for the new implementation that match the two versions.

7. Retries and ordering

The automatic retry of a step in case of a failure is key feature. You can control ordering to some degree by putting longer and longer delays when retrying a failing step. This technique is sometimes called exponential back-off, since the time between retries exponentially increase to ensure throughput of succesful jobs. The default retry delay is calculated as delay = retries^3 seconds.

If you want to stop retrying either return a step.Fail() or throw FailCurrentStepException.

Step execution is only orderes by an earliest execution time. If you need to control that step "B" execute before step "C". Then from step "A" spawn a step "B", and in step "B" spawn a step "C".

8. Micro Workflow and related concepts

Another way to gain conceptual insights into the framework, we explain why Micro Workflow is a good implementation fit to many concepts.

Micro Workflow as a Queue

You may not think of Micro Workflow as a queue since the step execution is unordered. Queue's are asociated with FIFO - First In First Out. A consequence of FIFO is that when queue elements can fail and retry, the FIFO property will stop the entire queue. For most real life scenarios this is unacceptable, hence most queues are in fact not FIFO.

Thus we can implement a queue as a a workflow with only one step.

Micro Workflow as a Job scheduler

The system can act as a job scheduler. A step can be scheduled for a certain time and re-executed again at a certain time. To ensure only one instance exist, use the Singleton attribute.

Micro Workflow as the 'outbox pattern'

The outbox pattern is an implementation strategy you often read about when dealing with events or distributed systems. It is a way to ensure that notifying other systems of a change happens in the same transaction as the change itself. The implementation is simply to insert a row into a queue that notifies the other system.

This is exactly a one-to-one match with a step in Micro Workflow.