Why Workflow Sucks

Many of us involved in the field of Workflow Automation and Business Process Management (BPM) have argued long and hard about where these two technologies overlap, where they are different which mathematical models to use, which standards are applicable to which part of the technology stack, and all that associated puff.

Well, these arguments and discussions are over; the demarcation lines have been drawn; the road ahead is clear.

The fact that Business Process Management has its roots in Workflow technology is well known - many of today's leading products are, in fact, evolutions of the original forms processing packages. So there is no longer a need to debate what is now a moot point.

But what has happened is that BPM has also changed. Rather than being an extension of workflow concepts, BPM is now seen as systems-to-systems technology exclusively used in the deployment of SOA solutions. I'm over simplifying things, I know, but it does seem that BPM is becoming an IT Technology solution as opposed to the business process solution it was meant to be. Somewhere along the way, one of the key elements in a business process - a person - dropped off the agenda. The fact that the majority of business processes (some 85% according to the analyst company Forrester) involve carbon-based resources was overlooked - think BPEL for a moment - doesn't the development of that particular standard tell you something about the general direction of BPM? But be warned, many vendors will tell you that their BPM products support human interaction, but what they are talking about will be simple work such as item handling and form filling - this is a long way from the collaboration and interaction management we will talk about below.

The problem stems from the fact that most Workflow products were flawed, and as a result, the problem in the gene pool has rippled through to the new BPM species. So what was wrong with workflow? It's quite simple when you think about it; most workflow products assumed that work moved from one resource to another. One user entered the loan details, another approved it. But business doesn't work like that.

This flawed thinking is probably the main reason why workflow was never quite the success most pundits thought it would be; the solutions were just not flexible enough, since the majority of processes are unsuited to this way of working. Paradoxically, it is the exact reason why BPM is so suited to the world of SOA and systems to systems processes. A rigid approach to systems processes is essential, where people are concerned; the name of the game is flexibility.

Why do we need the flexibility?

Let's take a simple analogy so that the concept is more easily understood.

Supposing you were playing golf; using the BPM approach would be like hitting a hole in one every time you tee off. Impressive - 18 shots, and a round finished in 25 minutes.

But as we all know, the reality is somewhat different (well, my golf is different) - there's a lot that happens between teeing off and finishing a hole. Ideally, about four shots (think nodes in a process) - but you have to deal with the unexpected even though you know the unexpected is very likely; sand traps, water hazards, lost balls, free drops, collaboration with fellow players, unexpected consultation with the referee - and so it goes. Then there are 17 more holes to do - the result is an intricate and complex process with 18 targets but about 72 operations.

As mentioned earlier, we have to deal with the unexpected. This is not just about using a set of tools to deal with every anticipated business outcome or rule; we are talking about the management of true interaction that takes place between individuals and groups which cannot be predicted or encapsulated beforehand. This is because Business Processes exist at two levels - the predictable (the systems) and the un-predictable (the people).

The predictable aspects of the process are easily and well catered for by BPMS solutions - which is why the term Business Process Management is a misnomer since the perceived technology only addresses the integration aspects - with the close coupling with SOA (SOA needs BPM, but the converse is not true) there is an argument for renaming BPM to Services Process Management (SPM).

The advent of BPEL4People isn't going to fix the problem either, all that will happen is the shortcomings of Workflow will be replicated and it will be as difficult and expensive to implements as it ever was - and anyone who has tried to put together a business case for buying SOA/BPM will know the entire proposition will be a non-starter.

Understanding the business processes exist at two levels (the Silicon and the Carbon) takes us a long way towards understanding how we solve this problem. The key point is to recognize that the unpredictable actions of the carbon components are not ad-hoc processes, nor are they exception handling (ask anyone with a Six Sigma background about exceptions and you'll understand very quickly what I mean). This is all about the unstructured interactions between people - in particular, knowledge workers. These unstructured and unpredictable interactions can, and do, take place all the time - and it's only going to get worse! The advent of Web 2.0, social computing, SaaS etc. etc., are already having, and will continue to have, a profound effect on the way we manage and do business.

Process-based technology that understands the needs of people and supports the inherent "spontaneity" of the human mind is the next logical step, and we might be tempted to name this potential paradigm shift "Knowledge Intensive Business Processes."

KIBPM falls into two main categories, which will probably merge over time, and the vendor that recognizes that potential will steal a march on the others. At the simplest level we have case management, and secondly, we have Human Interaction Management. I doubt there are many BPM products on the market today which will be able to meet this seismic shift in requirements - certainly those that rely on BPEL and SOA won't; what's more, any that have been in the market for longer than five years will need radical surgery to meet the coming challenge.

So why does workflow suck? It sucks because it made the fatal assumption that a business process was simply modeled as "a to b to c" - but business, as we all know, doesn't quite work like that. BPM succeeds because of the heritage these products have in the workflow world - but BPM sucks as well because it ignores the requirement to include people.