Now that the BPEL4People specification has finally been released, it is important to clarify exactly what it covers, and what it doesn't cover.

Let's start with something my fellow ebizQ writer Michael Dortsch said recently in his blog (12 April 2007), following some comments I made to an earlier post of his:

"I think I now think there are two classes of processes - task-driven and human-driven. I think I also now think that task-driven processes define and/or how tasks get done, often by IT systems and resources, while human-driven processes define and/or describe how people do things. Both are essential to successful business operations. However, they are very different from one another, and cannot likely be equally effectively addressed by any common set of processes or technologies."

Why are BPM experts like Michael making such a dramatic division?  Because all mainstream BPM languages, including graphical notations like BPMN, are based on carrying out steps in a pre-defined sequence - and this is not what happens when humans work together.

Turning to BPEL4People, it claims to support patterns for "human interactions".  However, in terms of the above division, these are human interactions in "task-driven" processes. In other words, what the specification authors mean by "human interactions" is interactions between humans and systems (H2S) - not interactions between humans and humans (H2H).

Let's take an example.  Suppose a document is escalated by Joe to Jane for approval - a typical application for BPEL4People.  What happens is this.  First, Joe tells the system "this document needs higher approval".  Then, the system works out that Jane should do it, and adds the approval task to her worklist.  Finally, next time Jane accesses her worklist, she sees a new document for approval.  There is no direct interaction between Joe and Jane at all.  Everything is mediated by worklists, or notification managers, the logic behind which is largely set in stone at the time the processes were originally implemented.

This is an entirely separate issue from dealing with interactions between humans and humans (H2H).  Such work is collaborative, innovative, and adaptive - and cannot be supported with a language such as BPEL, no matter how it is extended.  Let's take another example.  Suppose Joe uncovers a new opportunity with a client.  He knows that to realize the opportunity he will need support from others, including Jane.  He needs to start a new work process to deal with the opportunity - and this work process will involve a number of people, although at the start the only person Joe knows he will need is Jane.  Others will be included as they go along - similarly, they will define on-the-fly the various responsibilities, activities, and deliverables of each person.  If you know anything at all about BPEL4People, it will be immediately obvious that it could never be used to support such a work process.  Rather, H2H interactions are the domain of Human Interaction Management (HIM).

BPEL4People is related to HIM, in that both deal with humans, but the 2 approaches are complementary rather than competitive.  In fact, they are often closely tied together.  Let's take a third example - a system responsible for resolving faults in a telecommunications network.  Most of the faults are handled entirely automatically.  Some need minor manual involvement - for instance, to approve an engineer visit.  However, a small but significant percentage cannot be resolved in either way.  This last type of faults is the complex problems, those that humans need to work together to solve.  Typically, a solution may require several different people to collaborate - call centre staff, network specialists, service engineers, exchange staff, and the customer themselves - in order to determine the cause of the fault and find a way to deal with it.

Managing complex problems like this is where much of the money is spent.  A large telco may spend tens of millions of dollars per annum on such work - at least, they may if they have not yet adopted HIM techniques and tools.  With HIM, the solution is simple.  Most faults are handled via conventional workflow - defined using BPMN, BPEL (with or without BPEL4People), XPDL, or any other notation.  When it comes to a complex fault, a human using the workflow system passes the issue out to a Human Interaction Management System (HIMS) for resolution, choosing as a starting point what seems to be the most suitable template "Story" (i.e., work process).  The humans involved in the Story - who may well increase in number during its life - bring the issue to the point at which automated resolution is again possible; they evaluate the problem, agree a solution, and set it up.  Then the work is passed back to the workflow system, by invoking a suitable set of Web services to clear the fault.

When the complex faults are handled like this, the work becomes tractable.  It can be structured, managed, supported, and improved - just like the automated or semi-automated work required to resolve simpler faults.  When it comes to the bottom line, less money is spent, less time is taken, and both the telco and their customers are happier.

TAKE AWAY

Lots of things that happen in organizations simply do not match in any way the underlying "parallel flowchart" paradigm of a language like BPEL, no matter how you extend it. Look at the sort of things you personally do at work, for example. The "human interaction" patterns that the BPEL4People specification authors are concerned with are useful in integrating some forms of workflow, such as document approvals, into mainstream BPM.  It will be useful for many organizations to handle such patterns using the same BPEL-based technology that they use for fully automated processes.  However, these patterns are totally inappropriate for activity that is innovative, collaborative, and adaptive.

Today's attempts to bundle H2H process support into workflow systems or office applications are confusing apples with oranges, since they are attempting to impose on H2H interactions the wrong metaphor.  Humans trying to collaborate find it not only unhelpful but actually abhorrent to view their interactions with each other as a set of pre-defined steps carried out in sequence, and will either ignore, work around or subvert such bastardized pseudo-solutions.  They know there is a lot more involved: notions of responsibility, accountability, goal-directed communication, privacy, rework, prioritization, consensus, authority, and a lot more besides.

To deal properly with H2H interactions, you need appropriate techniques (HIM) and tools (the HIMS).  These techniques and tools are not hard to use, since they are nothing but a formalization of common sense - a direct reflection of what really happens.  This formalization comes from many years of research and practical experience, which has resulted in well-defined ideas, principles and patterns - ideas, principles and patterns that genuinely meet organizational and individual needs for control over H2H interactions.

Posted by keithhb in Business Process Management | Permalink | Comments (9)

In my role as a consultant, I was recently asked to help a large-scale safety-critical project with their fault management. Despite the use of design-by-contract techniques, that on smaller systems typically ensure a very low defect density, the sheer size of this system meant that the number of bugs was far higher than expected. As a result, the project has been struggling in various ways.

My recommendations had various aspects:

1. Strategic: the introduction of new code delivery processes based on Human Interaction Management principles;


2. Tactical: emergency measures to ensure that upcoming deadlines are met;


3. Operational: rationalization of how they are going about fault diagnosis.

The last of these approaches is what I want to discuss here - and in particular, I want to draw out how it relates to BPM.

When I say "fault diagnosis" I mean the engineering work that takes place in between a test failing and the start of coding to implement a solution, i.e., the detection of the corresponding bugs in the code, and the design of a solution. There are 2 related problems with the way my client is going about fault diagnosis on this project.

First, all the complex faults are handed to a small number of key individuals, who possess not only a high degree of skill but also extensive knowledge of the system. This means that they get better and better at fault fixing, while others do not improve. The project team is polarizing.

Second, even among this small cadre of experts there is no consistency imposed on the way they go about fault diagnosis. As a result, it seems likely that many faults are being mis-diagnosed: attributed to the wrong causes, only partially fixed, or fixed in such a way that new errors are introduced in other parts of the system.

As part of the solution to this operational problem, I created a generic checklist for "Software Fault Diagnosis". This consists of 4 stages (Symptoms, Scenarios, Sources, Solutions), each containing a number of steps. We are now customizing this checklist for the system in question, with the intention of giving it to each developer to put on their wall. It has 2 purposes: as an artefact capturing the knowledge currently in key individuals' heads (which can be maintained along with the system itself), and as a means of improving how everyone does fault fixing.

Interestingly, when I came to draw up the generic checklist, I assumed that it would be possible to find such an item for download somewhere on the Web. Yet a wide trawl of the research literature revealed nothing suitable - certainly nothing applicable to a large-scale, highly distributed system. I had to combine elements from many different sources to put it together.

What has all this to do with BPM?

In my last post I touched on the problems with verification of large processes designed using mainstream BPM applications. Put simply, currently there does not seem to be an equivalent in the BPM world for the various forms of testing deemed essential when software is constructed by any other means. Following this train of thought, there is a next logical question to ask.

TAKE AWAY

Suppose you have just introduced a large-scale process, built using a BPM suite, and the process is not working as expected. What tools do you have for finding the fault?

I will be very interested to hear how readers of this blog are going about it. From my own observations, I suspect that fault diagnosis in BPM-based systems is currently very immature. This may not be surprising when you consider that, even after over 50 years of commercial software, fault diagnosis in conventional software is not yet standardized - as evinced by my inability to find a standard procedure for it in the research literature.

However, at least with conventional software, there are well-known and proven techniques (such as binary search) and tools (such as interactive debuggers) for fault fixing. For fault diagnosis, as for testing, it seems that BPM is not yet delivering enterprise-strength solutions. This represents a serious business risk of which any current or prospective BPM user needs at least to be aware.

Posted by keithhb in Business Process Management | Permalink | Comments (1)

Abraham Maslow, creator of the hierarchy of human needs, famously said: "If the only tool you have is a hammer, you tend to see every problem as a nail." But what if you have a Swiss Army Knife?

This is the 12th and final article in my blog series on The Future of Programming. The series started back in March, and has addressed various topics including:

• Dealing with acronym hell


• BPM without a BPMS


• Managing outsourced application development


• Professional development for developers


• Quality planning


• Managing change in large-scale development

In this concluding article, I will discuss a theme that underlies all these topics.

When I entered the IT industry, I was trained to become not just a software developer, but a software engineer. What is the difference?

Engineering is fundamentally about techniques and tools. An engineering job consists essentially of assessing the situation, deciding what techniques and tools are suitable, then applying them. Hence, in order to be an engineer, you need not only depth of knowledge, but also breadth - a wide-ranging understanding of problem types and solution types.

I began my IT career in the late 1980s. At this time, object-oriented programming was just getting started. Some would say that it is still is! However, there were certainly fewer mainstream programming languages back then, fewer (and less powerful) development tools, and far fewer programmers. Also there were hardly any packaged applications for business use. As a result, it took a lot longer and cost a lot more to introduce new software into an organization.

As a result, it was more common for large-scale software projects to be conducted on quite a formal basis. Not only was the choice of tools and techniques more limited, making it easier to apply an engineering approach to software development, but also the stakes were higher. Despite this, many projects failed, of course, but at least there was recognition - fueled by the writings of thinkers such as Fred Brooks - that software development needed very careful handling if it was to succeed.

Now, in today's world, we have a plethora of new development tools - including BPM suites claiming to make it possible for business users to build their own enterprise software, and SOA suites claiming to make it possible for an organization of any size to share bits and pieces of code willy-nilly. As a result, many organizations are dispensing with the costly and time-consuming engineering processes, in favour of what appear to be the latest and greatest quick-fix solutions.

Well, why not, you may ask? I have 2 main fears about current development trends.

First, organizations seem to be confusing tactical and strategic solutions. A set of BPMN flowcharts may speed up your logistics process, or shave 1% from your invoicing costs. But this is no substitute for an organizational model, a model that is based on high-level business goals. It is easy to adopt short term point solutions that in the medium term fragment your enterprise IT architecture, and in the long term render it unmanageable. In particular, the true problems of SOA Governance are only just starting to emerge. BPM and SOA may in fact be driving a return to the integration nightmare that they promise to solve.

Second, whatever happened to testing? I recently heard a BPM Suite vendor speak proudly of a BPMN process they had implemented that contained 250,000 steps. This represents a vastly complicated piece of business software, so I asked how they had tested it. He replied that testing was unnecessary since their tools did not require the user to write lines of code (just to draw diagrams), and anyway their products contained simulation features if you really wanted to prove that an application worked as expected. He then went on to say how this approach must be OK, since other organizations are using their suite to build safety-critical applications.

This response made me blanch, and still sends shivers down my spine. What sort of world is being constructed using such tools? Whether created via diagrams or as text, software needs testing - and simulation is no replacement for structured, automated, exhaustive, regressive verification. Further, anyone who knows anything about the theory and practice of simulation will recognize immediately that simulation of a flowchart of 250,000 steps is itself a hugely complex and risky exercise.

TAKE AWAY

No matter what tools you use, it is highly dangerous to take the engineering out of software development - yet this is exactly what is happening with the emergence of new BPM and SOA tools.

If you are responsible for software development, I suggest you ask yourself a simple question. Do I care what happens 12 months down the line?

If not, then you can probably ignore much of what I have written in this series.

Otherwise, I suggest you stop and take stock. Remember that much of what you read online and in magazines is written by people with a vested interest in getting you to adopt certain new commercial products. It takes only common sense to see that the practice of software development is nothing more and nothing less than engineering - and engineers in other fields are very cautious indeed about using new techniques and tools to build big things. Why should software be any different?

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

This is the eleventh in a blog series on the Future of Programming. Last time I started to explain the diagram of a "collaborative process pattern" that helps to bring large software projects under safe management control, by structuring the way you handle the continual revision to the development process that inevitably arises when different people are responsible for different aspects of the design. Please refer back to previous articles in this series for details. This time I will finish discussing (for now) how the diagram can help you to manage large software developments, by explaining the third and final perspective from which to understand it - the solution point of view - and how to put the diagram into practice as a means of managing ongoing operations.

From a solution perspective, the Role Activity Diagram is divided into two collaborative transactions - one at overall design level, and one at domain design level. Each transaction is bounded above and below by a 3-way interaction. In each case, the interaction at the top represents an exchange of requirements while the interaction below represents an exchange of solutions.

Both the requirements exchanged in the top interaction and the solutions exchanged in the bottom interaction will involve complex, compound objects. A solution interaction, for example, may pass across:

• The final problem statement, now fully developed
• The objects that form the solution, contributed by different Roles
• The models and tests that justify the solution
• A map of the dependencies between all these objects (for use in tracking the impact of changes) and a related approval chart, showing who has approved what. (Both these can be maintained automatically. Whenever a participating Role submits a new version of one of their deliverables, all the approvals that are dependent on the new deliverable are removed and new approval activities are triggered. Processes and object relationships interact - as long as they are both represented.)

In between the upper and lower interactions, each participating Role will carry out various different activities, in order to provide its own part of the solution. The Role Activity Diagram shown here is simplified and omits these activities, as well as the phases into which the activities will be broken during the sub-project. Why?

All we need at this stage is to represent a consensus from the initial meeting on how to proceed with the sub-project. We are not seeking to prescribe exactly what everyone will do to meet the requirements placed on them - just to establish a common ground on how the issue as a whole will be dealt with. What we are interested in initially is to make it clear to all parties what their responsibilities are to the process as a whole. We need to show only who is to do what, how they will deliver their outputs, and what impact each person's work has on the rest of the process.  The diagram effectively models not only the work required, but also the executive and management controls required over the sub-project.

Once the work gets going, it is inevitable that changes will occur to the process shown in the Role Activity Diagram. In particular, the activities in the sub-project will be extended, and broken into distinct phases, and during the course of the sub-project the elements of the diagram will be extended further as required. The project participants will make agreement depictions at overview or detailed levels that not only refine the process, but also allow all concerned to share the consensus reached about next steps.  In particular, the deliverables may change - perhaps certain ones will be altered to include only partial functionality, and the deficit remedied by other means (or just agreed with the customer as a concession against requirements).

It is here that day-to-day management control can be applied, in order to ensure that the process as it unfolds in operation takes place safely, in accordance with any applicable regulations, and with all parties committed to the same manner of proceeding.

Moreover, it will be possible to do this at either level without contravening the executive control applied from above, since it is clear from the diagram how this control is applied - via the application of the REACT pattern. Should the process participants at any time decide that this application was inappropriate, this change can be described in a simple way via a new Role Activity Diagram, and submitted to the appropriate Role for approval.  For instance, suppose that in a particular domain - safety engineering, say - that the Domain Design Authority feels it sensible to share some of his or her Research activity with the Designers of their part of the solution. After discussion with the Domain Sponsor and Designer(s) in question, the Role Activity Diagram can be altered to show the proposed process change, and sent to the Domain Sponsor for official approval.

There is a lot more one can say about such changes, but for now all we need to do is note that the diagram provides a simple means of achieving shared understanding on a new way forward, and of ensuring that this way forward is safely managed.

TAKE AWAY

Perhaps the primary concern for anyone charged with a large-scale software development should be management of faults, issues and other things that in some way force change to the current plan.  It is a law of nature that such changes happen during software development.  With small-scale software development, agile techniques suffice.  With large-scale software development, my experience, both first- and second-hand, is that they don't.  Having separate teams and/or sub-projects operating concurrently changes the situation fundamentally, and you need something more - something industrial strength - in order to cope.

In the last few articles in this series I have outlined a technique that can help with large-scale software development.  In the next - and concluding - article in this series, I will offer some observations on general emerging trends in enterprise software development.  In particular, I will discuss how the adoption of new BPM/SOA tools is bringing with it some extremely dangerous practices.  To find out what they are, and how to avoid them, tune in next time.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)