Regular readers of this blog will know of my view that the next significant step in both IT and management is to improve the way that people do collaborative knowledge work (what I call "interaction work"). You can find all sorts of evidence for this view on the Human Interaction Management Web site. However, most people know from their own experience that action is needed, and the sooner the better. In particular, working hours are getting out of control, a trend that is not helped by the current expectation that everyone is "always on" via their mobile phone.

A typical factor in people's inability to control their workload is the necessity to use inefficient workplace software, the main culprit being email. Across the board in industry there are major problems resulting from use of email. It is not at all clear when emails should be sent, what they should contain, who should be included, the validity of requesting actions from people, who has committed to do what, the correlation between different email streams, the accuracy of data included, etc. Almost daily, this lack of clarity causes material problems in organizations of every kind.

Problems such as email usage are complex to resolve, relating as they do to process awareness, new approaches to management, new forms of software, and more. Hence solutions for such problems can be expensive to implement in large organizations. How can one justify the expense? In other words, what is the basis of a business case for the introduction of more efficient human collaboration?

It is necessary to find a quantitative means of evaluating the impact. One can list ad nauseam aspects of organizational life that will be improved by rationalizing the way people work together, but the board have a duty to consider the bottom line in any major decisions they make. However swayed they may be personally - e.g., from experience of mobile phone calls at ungodly hours - they will not be acting with due diligence if they approve a programme that has no means to demonstrate Return On Investment (ROI). So it is necessary to find a metric that is applicable to human collaboration.

This is non-trivial, since there is a difference between efficiency and effectiveness.

When measuring the impact of process improvement in mechanistic areas such as transaction handling or manufacturing, the outputs of the process are probably going to remain similar after the changes have been implemented. The aim is simply to produce these outputs quicker and cheaper. In other words, one is aiming for increased efficiency.

The same is not at all true for human collaboration. Free people up by helping them work better, and they will deliver more value to the organization. Once people are not struggling to keep afloat, they can help steer. In other words, improving interaction work delivers increased effectiveness. However, in advance one cannot always predict what form the increase will take. Take sales, for example. One might expect a salesperson who works better to make more sales, but it is quite possible that the actual improvement will be customer relationships that are longer-lasting, something that cannot be measured until enough time has passed.

So what metric should one use when proposing a programme to improve human collaboration in the organization?

TAKE AWAY

The simplest metric for human collaboration is very simple indeed. Work out how much you pay people per hour. Then count how many hours they are spending at work, taking care to include work done out of the office. If your staff start taking less time to do their work, you are getting better value for money.

This applies even if you don't pay people overtime. There is a huge hidden cost to working long hours. Tiredness and stress not only make people miserable, but reduce their contribution to the organization and increase "churn" (the frequency at which staff leave the organization altogether). These negative impacts are at least partially offset by paying people decent overtime - and if you don't pay people overtime directly, you can be sure that your organization is paying the price somehow.

So a first step for a programme to improve interaction work in your organization is to find out how much time in each day people are taking to do their work. Cost this time based on salaries, and aim to reduce the total by a specific amount - say 1 hour per day per person. This effectively gives you a budget for the interaction work improvement programme - spend any less than this in total, and you will have delivered ROI.

Of course, this is using the narrowest of measures - a measure that takes no account of the significant improvements you can expect in effectiveness, which as discussed above will be the main benefit of the programme. However, the aim here is simply to get the programme approved by the board. Once you get going, material impacts of all kinds will become evident. It will be much easier to get approval for your second interaction work improvement programme than for your first!

Posted by keithhb in Business Process Management | Permalink | Comments (0)

Readers of my last post on the knowledge bus may be interested in James Taylor's response, to which I added an explicatory comment.

A case study illustrating use of the knowledge bus will be published in my bptrends.com column "Human Processes" in March. When it appears, I'll link to it from this blog.

The case study can be found here:

"What is the next major step in enterprise IT? Keith Harrison Broninski has a fascinating perspective. He argues convincingly that there will be a shift in emphasis from server-side automation application to client-side human interaction. Read his Column for the details of why he’s convinced this change will occur. "

Posted by keithhb in Business Process Management | Permalink | Comments (0)

This post is the third and concluding part of my predictions for 2008. In the last 2 posts, I have described how the flood of new technologies in recent years is causing some apprehension among business people. Many remember how hard it was to measure ROI from (for example) ERP - and some remember implementations in which ROI was conspicuous by its absence. How much more complex will it be to measure ROI on BPM+SOA+BRM+CEP+BAM+BI+Mashup+add_your_new_type_of_product_here?

In particular, it is not yet at all clear what is involved in the administration of these new technologies. For more on this, you may be interested to read my latest column on BP Trends, in which I discuss SOA governance from a human perspective.

The crux of the problem is that major software vendors are addressing a technology stack, not a business stack. Recent acquisitions by hard-core technology companies such as Oracle and Sun have reinforced the popular impression that new technologies are aimed squarely at improving core IT operations, not wider business operations. To many business people, technology vendors fail to supply anything that might address their key concerns.

In particular, any disinterested observer of global business must notice immediately that the main thing happening to organizations of all sizes and types is outsourcing. Mom-and-pop-shops are not just outsourcing their book-keeping but also their services, traditionally a core offering of a small organization. Blue-chips are outsourcing their manufacturing, warehousing, logistics, ... you name it. HP has been outsourcing printer design for many years now.

When talking to outsourcing vendors while in India recently, I was struck by the maturity of their business approach. They have come to understand very well what works and what doesn't, and the argument for working with such partners is becoming almost irresistible. So for many business people charged with delivering cost savings and business improvement, a key way forward is Business Process Outsourcing. This means that the main activities required in 2008 are as follows:

1. Process Architecture. Chunk up the organization into a network of co-operating processes - strategic, tactical and operational - and decide which of these should be the current focus.
2. A Wider Vision Of Choreography. Work out the interactions between these processes, and hence which ones can safely be outsourced. What shall we do? What shall our partners do?
3. Total Cost of Ownership. For those processes that are left over - and only those processes that are left over - assess the validity of streamlining them an IT-based implementation. In particular, how can we optimize the IT operations necessary to administer new solutions?

TAKE AWAY

For many people, 2008 will be about process architecture. This is not the same as process mapping. Architecture tells you what processes you have and what the relationships between them are - mapping tells you the steps in a particular process. There are various techniques for process architecture, but the only one based on the realities of networked organizational activity is Riva - though use of Riva does require some care.

The true value of process architecture for many organizations will be that it leads on to a wider vision of process choreography, a vision encompassing a lot more than technical notations such as WS-CDL. In the end, why implement what you don't own? For more details, take a look at the GOOD methodology.

And when you finally decide to implement a process, how do you do it as cheaply as possible? Anyone who has been around a while knows that Total Cost of Ownership depends fundamentally on the ongoing administration costs - and current IT administration frameworks do not go far enough. ITIL tells you what you must do, and COBIT tells you how well you are doing it, but how do you define and implement your processes for IT administration?

Since IT administration is carried out by humans, the missing link here is Human Interaction Management, facilitated by a new breed of lightweight and low-cost tools: the Human Interaction Management System.

2008 may well be the year in which business people start getting true value from technology - by using less of it.

PS: If you would like to hear further explanation of these ideas, I discussed them during a recent ebizQ podcast with Elizabeth Kratz and other writers for this site.

Posted by keithhb in Business Process Management | Permalink | Comments (2)

This is the second in a series of 3 predictions, of ideas that I believe will be publicly accepted by the end of 2008. Last time I discussed how "The new wave of IT solutions is far too complex for most people". This time I'll go into some more detail.

In particular, my second prediction - that people will come to see how "The new wave of IT solutions raises as many problems as it solves" - is based on a phrase that 10 years ago one used to see a lot more than nowadays, namely Total Cost of Ownership (TCO).

The term TCO came into widespread popular usage for IT purposes around the mid-1990s, when Oracle Corporation and Sun Microsystems in particular started propounding the advantages of the "Network Computer" - a true thin client that possessed no hard disk, only an Internet connection. However, it took a long time for the advance of the Web to bring with it software that can support thin client solutions without reducing the user experience unacceptably - AJAX-based applications, for example. Even now, nearly all computing devices still possess a hard disk, and in my view will probably do so for the forseeable future.

So vendors no longer bandy about TCO as much as they did, since they see less mileage to be gained by doing so. Instead, IT customers are starting to take on these concerns. Faced with proposals from vendors, and analysis on sites such as ebizQ, that suggest the modern enterprise needs not just BPM, but BPM+SOA+BRM+CEP+BAM+BI+Mashup+add_your_new_type_of_product_here, organizations are starting to worry very seriously about administrative overheads. Which is not surprising, since:

• These products are new.


• They are complex.


• There are a lot of them.


• Hardly anyone understands how to use them properly.

TAKE AWAY

Part of the reason that corporate decision-makers are deciding to focus on the management aspects of process-orientation (rather than the technology aspects) is due to the rise of Business Process Outsourcing (BPO). Why implement what you may not end up owning?

The outsourcing industry has matured enormously in recent years. Talking to outsourcing vendors while in India recently, I was deeply impressed by their realistic grasp of what works and what doesn't in outsourcing, and how customers can get the best from low-cost commodity labour.

Further, the outsourcing model has spread to the point where it is no longer simply a matter of cost reduction (if it ever was). The rise of "homeshoring" attests to this.

However, the increased interest in outsourcing raises questions of its own. Next time I will address my third prediction - that people will recognize during 2008 how "There are fundamental business problems that need help from IT but that are not addressed at all by the new wave of IT solutions" - and discuss where the market will turn for answers to these questions.

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (2)

Happy New Year. As as traditional at this time of year, I will make a few predictions for the IT landscape in 2008.

Or rather, I will predict some acknowledgements - state some ideas that I believe will be publicly accepted by the end of 2008. These are about what might be called the "new wave of IT": the host of new techniques and technologies that have recently become contenders for a place on the enterprise backbone. SOA and BPMS are obvious candidates for inclusion, along with associated acronyms such as CEP, BRM, mashup, and so on.

So here are the acknowledgements that I believe will happen in 2008:

1. The new wave of IT solutions is far too complex for most people


2. The new wave of IT solutions raises as many problems as it solves


3. There are fundamental business problems that need help from IT but that are not addressed at all by the new wave of IT solutions

Over the next few days I'll take these in turn, starting with:

The new wave of IT solutions is far too complex for most people

If there is a single subtext to the various discussions I have with business people, it is confusion. And this is hardly surprising. Consider BPM, for example, and let's look again at Ismael Ghalimi's original definition of "BPM 2.0" from Feb 2006:

BPM 2.0 is not for non-technical business analysts. Never should have been, never will, and nobody should care.
http://itredux.com/blog/2006/02/01/bpm-20/

Despite this frank admission from a founding figure in the BPM space, and the efforts of certain commentators to explain that BPM is first and foremost a management approach, analysts and vendors continue to present BPM as a set of tools aimed at business users. Presumably this is in the hope that the take-up of such tools will thereby be helped, by broadening the potential market. However, if anything take-up has been harmed by this approach - most business people I talk to are apprehensive about BPM tools, even if they are drawn to BPM as a means of organizational transformation.

Let's face it head-on: BPM tools are simply a new form of programming, and as with any other new form of programming, it is not yet well understood how to use them safely. In particular, mainstream programming aids such as automated testing and static analysis are not even available for BPM tools.

It is no different with SOA, CEP, BRM, mashup, and so on. In the end, these are all techie gadgets, and the first step towards making good use of them is to recognize that.

TAKE AWAY

The SOA analyst firm ZapThink, in their own predictions for 2008, write:

many organizations are still struggling with the business case for SOA, or even worse, have made a wrong turn or reached some kind of impasse with their SOA initiatives
http://www.ebizq.net/news/8772.html

It's hardly surprising. There are new business ideas emerging originally from IT, and resulting new software tools aimed at business users, but the larger incumbent software vendors are still busy trying to sell technologies. As a result, business users are either suspicious or struggling (or both) - and in 2008 we may well see a backlash.

Tune in to the next post for some more home truths about the new wave of IT. Welcome or not, you should consider these ideas before trusting your career, and your organization's livelihood, to the new wave of IT :-)

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (0)

Christmas is coming so it seems appropriate to return to some reflections I made a few weeks ago, on parallels between enterprise technology and church architecture (see Mediaeval middleware and Stress-Oriented Architecture). And it seems I am not the only person making this connection.

Frits Bussemaker, founder and chairman of the Dutch BPM-Forum, wrote to me last week to ask whether I had an answer to a question he posed back in July, via his BP Trends column Gaudi & Gravity. Frits reflects on Gaudi's astonishing architectural masterpiece, La Sagrada Familia in Barcelona. Anyone who has seen this building cannot fail to be astonished - even today it is like nothing else. Personally I think it very beautiful. Images don't do La Sagrada Familia justice, especially now that construction works aimed at completing the building fill the background with cranes (Gaudi himself never managed to build the entire cathedral), but here is an aerial photo from before works started:

By contrast, George Orwell hated La Sagrada Familia. I have just finished re-reading his "Homage to Catalonia", and was surprised by his description of Barcelona's cathedral as "one of the most hideous buildings in the world". He writes, "I think the Anarchists showed bad taste in not blowing it up when they had the chance" (p.214, Penguin edition). Orwell really could be an old grouch sometimes.

Frits discusses how Gaudi "used the law of gravity to model and create this impressive and complex building in the real physical world", and asks:

Is there an equivalent for gravity in our imaginary world of IT, organization, and processes? If there is such an equivalent then we probably would have some very simple principles to shape our complex imaginary world.

Frits posits an answer as follows:

I suspect that the business process is the rope in Gaudi’s model and entities like competitors, shareholders, regulations, and, most important of all, clients will pull an organization into its optimum shape.

I agree with Frits that the "rope" binding together stakeholders in an organization is processes. However, I would add that these processes are not the kind that I call "mechanistic", but rather the kind that I call "human-driven".

Mechanistic processes are like the services of a building (electricity, heating, plumbing, etc). They seem vital, but you can manage without them if you have to. Orders can be taken, invoices created, payments processed and goods delivered by hand if necessary.

In fact it is the human-driven processes (collaborative, adaptive, innovative human work) of an organization that are the infrastructure. Imagine an organization in which no-one was making sales, dealing with customers and suppliers, assigning staff, managing work, signing off payroll, resolving issues, ... - everything would fall apart immediately!

TAKE AWAY

One of my first white papers, back in 2004, reminded people that:

An organization in which people interacted only via scripts loaded into machines could not think, could not respond, could not change and could not possibly provide effective support for its customers.

The trouble is, without Human Interaction Management (HIM), it is not obvious what such interactions consist of.

There is growing recognition that in today’s wired, globalized world, most routine work is being standardized, outsourced and automated. In other words, all that organizations have left to compete on is the interactive knowledge work (what I call “interaction work”) that is left over. HIM provides a means of doing such work more efficiently, and managing it better.

HIM also offers the opportunity to overcome the inherently fickle nature of Internet trading by binding your customers into shared, long-lived, collaborative processes. This is the route towards what Pine and Gilmore (authors of “The Experience Economy”) call the “fifth economic offering”: transformations.

Returning to Christmassy matters, one can make an analogy with modern business and the progress of a pantomime. We are currently approaching what is called the "transformation scene" - a moment when the protagonists get themselves out of trouble by discovering some device that allows them to enter a new and exciting world. The trouble is "Asia, Automation and Abundance". The device is Human Interaction Management. And the new and exciting world is what Information Age herald as follows:

A new generation of people-centric collaborative information management tools is set to produce the first fundamental advances in personal productivity since the arrival of the spreadsheet.

A very Merry Christmas and Happy New Year to you all.

Posted by keithhb in Business Process Management | Permalink | Comments (2)

This morning a colleague pointed me to a blog post by Dion Hinchcliffe, Leveraging Web 2.0 for business growth. Hinchcliffe, like many other pundits at the moment, paints a rosy picture of the new world of work that will be enabled by Web 2.0 technologies. The trouble is, this new world will only be rosy for a very few people indeed.

Hinchcliffe has picked up on the analyst firm McKinsey's term "tacit interactions". This term was described at length in their article "The next revolution in interactions", published in the McKinsey Quarterly, Q4 2005. The authors use the term to describe a particular form of collaborative knowledge work:

the searching, monitoring, and coordinating required to manage the exchange of goods and services

They go on to write:

Currently, jobs that involve participating in interactions rather than extracting raw materials or making finished goods account for more than 80% of all employment in the United States. And jobs involving the most complex type of interactions - those requiring employees to analyze information, grapple with ambiguity, and solve problems - make up the fastest-growing segment. This shift toward more complex interactions has dramatic implications for how companies organize and operate.

The authors conclude:

The shift from transactional to tacit interactions requires companies to think differently about how to improve performance - and about their technology investments. Companies can again create capabilities and advantages that rivals can't easily duplicate.

I agree whole heartedly with all this, and often quote the above words in support of my own work on Human Interaction Management (HIM). So it is good to see other writers quoting it too! However, when it comes to implementing the "shift from transactional to tacit interactions", Hinchcliffe (along with many other writers in love with all things 2.0) is looking in the wrong place.

Hinchcliffe, for example, describes the new wave of BPM tools as a means for people to implement their own lightweight process support for tacit interactions. However, consider this quote, taken from the article defining "BPM 2.0" to which he himself links:

BPM 2.0 is not for non-technical business analysts. Never should have been, never will, and nobody should care.

Indeed. BPM is for technicians, and this applies whether you are talking about version 1.0, version 2.0 or version 99.0. And it's not me saying this, but the software vendor (Intalio) who defined the term "BPM 2.0"!

So what about all the business people whose work is all about tacit interactions, but who could not reasonably describe themselves as "technical business analysts"? If we take Hinchcliffe at his word, this community is excluded by definition from the new and exciting Web 2.0 world. And I should think that this community includes about 99% of knowledge workers.

TAKE AWAY

The new world of work is exciting. Too exciting to leave it all to the techies! Web 2.0 tools are powerful levers for business growth, but efficient and effective use of them requires support for "tacit interactions" - i.e., a new form of business process.

I call this new form of business process "human-driven" - a term that should not be confused with the "human-centric" processes that are often discussed by analysts such as Forrester and are now supported by various BPM tools. Human-centric processes allow humans to interact with machines, typically to enter data or make low-level decisions, and have become mainstream. Human-driven processes are another thing entirely - they are a higher-level type of beast, and support for them is still emerging.

In particular, human-driven processes include interactions between humans and other humans, which requires new ideas and new tools. If you want to be part of the new world of work, BPM will not help you (whatever the version) - what you need is HIM.

Posted by keithhb in Business Process Management | Permalink | Comments (0)

This blog has often made the point that Business Process Management is - or at least, should be - about more than automation. In fact, automation is probably the last place you should be looking, not the first.

My general claim is that the value to be gained from process management lies not just in automating the "mechanistic" processes tackled by current mainstream BPM tools, since these are but the tip of the iceberg. The most interesting and important work carried out in any organization, the body of the iceberg, consists of "human-driven" processes carried out by people collaborating and innovating. It may be simpler to stick with mechanistic processes, since software vendors on all sides are pushing BPMN and BPEL. However, if you want true ROI from BPM, you will need to adopt new ideas and new tools, ideas and tools that are more suited to interactive knowledge work.

Consider, for instance, the interesting analysis of BPM in logistics by Roger Whitehead. The analysis was originally published a couple of years ago (July 2003) in Roger's "Looking Askance" column on the now-defunct BPMG Web site - it is now available here. Roger looked at how money a corporation could expect to save by implementing BPM software, and his conclusion was: in most cases, not much.

The particular example he chose to illustrate this view was of logistics costs, based on the 2003 report on supply chain performance from the Grocery Manufacturers of America. Logistics, as readers of this blog are no doubt aware, is a classic focus area for BPM, since most enterprises hope to automate it as far as possible in imitation of market leaders such as Dell and Wal-Mart. Here is Roger's summary:

I can see perhaps five areas of [logistics] cost that might be amenable to business process automation. They are:

1. planning, forecasting and scheduling (including inventory management) - 2.5 per cent
2. information processing (including e-commerce) - 1.8 per cent
3. customer service - 2.7 per cent
4. procurement management (or "buying", as we used to call it) - 1.1 per cent
5. plant warehousing - 7.4 per cent.

I have not included distribution centre costs, since these are mainly to do with goods handling. For some reason, this seems not to be an area where people use BPM much, if at all.

Together, the five areas listed above account for 15.5 per cent of the responding companies' total supply chain costs. That equates to just over 1 per cent of their net sales (15.5 per cent of 7.4 per cent). Since none of these companies is running at a loss, this is 1 per cent or less of their cost of sales.

Many of the activities under these five headings will already have been streamlined and computerised. The scope for cost cutting will therefore be less than in a 'green-field' situation. We could possibly reduce their cost by an average of a tenth, if we're lucky. That is roughly 0.1 per cent of these companies' total costs.

This is not going to cut the mustard. Something that might cut less than 0.1 per cent of a company's costs is not going to head anyone's list of favourite cost-reduction programmes. These companies would be looking to claw back elsewhere in redressing the 12 per cent increase in their costs since 2001.

Some companies are, of course, seeing greater cost savings than this from BPM. However, the general point Roger makes is that many enterprise activities "already have been streamlined and computerised", if only via implementing ERP packages during the 1990s. Proponents of SOA and BPM are currently struggling to convince boards that exposing the functions of these packages as Web services, and then hooking the functions up into new combinations via BPEL, is going to make any significant difference to running costs.

But why is the debate all about cost saving? Neither commercial nor government organizations have it as their mission to spend as little money as possible. What they are there to do is add value for customers. It is in adding value that enterprises have the chance to see true ROI from SOA and BPM, not in cutting costs. So the question becomes: are you more likely to add value by streamlining an existing set of routine activities, or by enhancing the way in which people work together, innovating as necessary, in order to delight your customers?

To help answer this question, let's take an example modern enterprise, a widget manufacturer that deals via the Internet both with suppliers and with customers, and ask: From what does their revenue arise?

Does it arise from their widget assembly operations? From the logistics activities that supply parts to these operations, warehouse the finished products, then deliver them to customers? From the transaction handling of payables and receivables? From the management of after-sales tasks such as returns management and servicing?

In fact, none of these contribute to revenue - they are all costs. For this reason, many organizations are now outsourcing all of the above in order to gain economies of scale. The revenue of the business derives from quite different sources; namely the collaborative, innovative human work required to conceive and design products, investigate markets, open up channels, make deals, resource activities, finance the enterprise, structure the organization, and manage all of this on an ongoing basis.

TAKE AWAY

All this can be summarized by saying that mechanistic processes are about increasing efficiency, while human-driven processes are about adding value. If your organization is commercial, your mechanistic processes are about saving money, while your human-driven processes are about making money.

So where should you be looking, if you want to get ROI from BPM or SOA? At mechanistic processes, with the hope of shaving a small percentage from your running costs? Or at human-driven processes, with the intention of achieving a transformation in the value you offer customers?

If it's the latter, then be careful. On every side, incumbent workflow and BPM vendors are offering re-badged versions of legacy products that claim to handle human-driven work, typically via a superficial integration with office software such as Word and Excel. There are also a few startups throwing hats into the ring, with claims that they can transform your working life by applying a BPMN template to lists of meeting actions. But treating human-driven processes as mechanistic will only damage them, not improve them. People actively dislike and resent being treated as cogs in a machine! Try to make your high-value staff into slaves of a new form of task list, however cleverly packaged, and you will get nothing but workarounds and backlash.

It is very tempting for managers to think that they can organize the interactions of knowledge workers in the same way that they organize the interactions of computer systems. However, attempts to do this will end up costing you very dearly. A task sequence, however described, is far too restrictive for high-value human work: there is no scope for everyday occurrences such as returning to previous versions of a document (or set of figures), revisiting a previous activity (or part of it), starting an activity (or part of it) before all the inputs have been received, and so on. Further, there is much more to knowledge work than can be captured in any task sequence: goals, responsibilities, commitments, and all sorts of other information that is fundamental to the success of the work but has no direct relation to tasks. Just to take one example, many human-driven processes require players whose contribution is not strictly functional - think RACI.

Rather than office applications with workflow lipstick, what you need is a new form of process management, that is based not on task lists, but on the true nature of human-driven work. Only in this way will you realise the true potential of business process management - and help your organization become a leader, rather than yet another follower.

Posted by keithhb in Business Process Management | Permalink | Comments (0)

Now that the BPEL4People specification has finally been released, it is important to clarify exactly what it covers, and what it doesn't cover.

Let's start with something my fellow ebizQ writer Michael Dortsch said recently in his blog (12 April 2007), following some comments I made to an earlier post of his:

"I think I now think there are two classes of processes - task-driven and human-driven. I think I also now think that task-driven processes define and/or how tasks get done, often by IT systems and resources, while human-driven processes define and/or describe how people do things. Both are essential to successful business operations. However, they are very different from one another, and cannot likely be equally effectively addressed by any common set of processes or technologies."

Why are BPM experts like Michael making such a dramatic division?  Because all mainstream BPM languages, including graphical notations like BPMN, are based on carrying out steps in a pre-defined sequence - and this is not what happens when humans work together.

Turning to BPEL4People, it claims to support patterns for "human interactions".  However, in terms of the above division, these are human interactions in "task-driven" processes. In other words, what the specification authors mean by "human interactions" is interactions between humans and systems (H2S) - not interactions between humans and humans (H2H).

Let's take an example.  Suppose a document is escalated by Joe to Jane for approval - a typical application for BPEL4People.  What happens is this.  First, Joe tells the system "this document needs higher approval".  Then, the system works out that Jane should do it, and adds the approval task to her worklist.  Finally, next time Jane accesses her worklist, she sees a new document for approval.  There is no direct interaction between Joe and Jane at all.  Everything is mediated by worklists, or notification managers, the logic behind which is largely set in stone at the time the processes were originally implemented.

This is an entirely separate issue from dealing with interactions between humans and humans (H2H).  Such work is collaborative, innovative, and adaptive - and cannot be supported with a language such as BPEL, no matter how it is extended.  Let's take another example.  Suppose Joe uncovers a new opportunity with a client.  He knows that to realize the opportunity he will need support from others, including Jane.  He needs to start a new work process to deal with the opportunity - and this work process will involve a number of people, although at the start the only person Joe knows he will need is Jane.  Others will be included as they go along - similarly, they will define on-the-fly the various responsibilities, activities, and deliverables of each person.  If you know anything at all about BPEL4People, it will be immediately obvious that it could never be used to support such a work process.  Rather, H2H interactions are the domain of Human Interaction Management (HIM).

BPEL4People is related to HIM, in that both deal with humans, but the 2 approaches are complementary rather than competitive.  In fact, they are often closely tied together.  Let's take a third example - a system responsible for resolving faults in a telecommunications network.  Most of the faults are handled entirely automatically.  Some need minor manual involvement - for instance, to approve an engineer visit.  However, a small but significant percentage cannot be resolved in either way.  This last type of faults is the complex problems, those that humans need to work together to solve.  Typically, a solution may require several different people to collaborate - call centre staff, network specialists, service engineers, exchange staff, and the customer themselves - in order to determine the cause of the fault and find a way to deal with it.

Managing complex problems like this is where much of the money is spent.  A large telco may spend tens of millions of dollars per annum on such work - at least, they may if they have not yet adopted HIM techniques and tools.  With HIM, the solution is simple.  Most faults are handled via conventional workflow - defined using BPMN, BPEL (with or without BPEL4People), XPDL, or any other notation.  When it comes to a complex fault, a human using the workflow system passes the issue out to a Human Interaction Management System (HIMS) for resolution, choosing as a starting point what seems to be the most suitable template "Story" (i.e., work process).  The humans involved in the Story - who may well increase in number during its life - bring the issue to the point at which automated resolution is again possible; they evaluate the problem, agree a solution, and set it up.  Then the work is passed back to the workflow system, by invoking a suitable set of Web services to clear the fault.

When the complex faults are handled like this, the work becomes tractable.  It can be structured, managed, supported, and improved - just like the automated or semi-automated work required to resolve simpler faults.  When it comes to the bottom line, less money is spent, less time is taken, and both the telco and their customers are happier.

TAKE AWAY

Lots of things that happen in organizations simply do not match in any way the underlying "parallel flowchart" paradigm of a language like BPEL, no matter how you extend it. Look at the sort of things you personally do at work, for example. The "human interaction" patterns that the BPEL4People specification authors are concerned with are useful in integrating some forms of workflow, such as document approvals, into mainstream BPM.  It will be useful for many organizations to handle such patterns using the same BPEL-based technology that they use for fully automated processes.  However, these patterns are totally inappropriate for activity that is innovative, collaborative, and adaptive.

Today's attempts to bundle H2H process support into workflow systems or office applications are confusing apples with oranges, since they are attempting to impose on H2H interactions the wrong metaphor.  Humans trying to collaborate find it not only unhelpful but actually abhorrent to view their interactions with each other as a set of pre-defined steps carried out in sequence, and will either ignore, work around or subvert such bastardized pseudo-solutions.  They know there is a lot more involved: notions of responsibility, accountability, goal-directed communication, privacy, rework, prioritization, consensus, authority, and a lot more besides.

To deal properly with H2H interactions, you need appropriate techniques (HIM) and tools (the HIMS).  These techniques and tools are not hard to use, since they are nothing but a formalization of common sense - a direct reflection of what really happens.  This formalization comes from many years of research and practical experience, which has resulted in well-defined ideas, principles and patterns - ideas, principles and patterns that genuinely meet organizational and individual needs for control over H2H interactions.

Posted by keithhb in Business Process Management | Permalink | Comments (9)

In my role as a consultant, I was recently asked to help a large-scale safety-critical project with their fault management. Despite the use of design-by-contract techniques, that on smaller systems typically ensure a very low defect density, the sheer size of this system meant that the number of bugs was far higher than expected. As a result, the project has been struggling in various ways.

My recommendations had various aspects:

1. Strategic: the introduction of new code delivery processes based on Human Interaction Management principles;


2. Tactical: emergency measures to ensure that upcoming deadlines are met;


3. Operational: rationalization of how they are going about fault diagnosis.

The last of these approaches is what I want to discuss here - and in particular, I want to draw out how it relates to BPM.

When I say "fault diagnosis" I mean the engineering work that takes place in between a test failing and the start of coding to implement a solution, i.e., the detection of the corresponding bugs in the code, and the design of a solution. There are 2 related problems with the way my client is going about fault diagnosis on this project.

First, all the complex faults are handed to a small number of key individuals, who possess not only a high degree of skill but also extensive knowledge of the system. This means that they get better and better at fault fixing, while others do not improve. The project team is polarizing.

Second, even among this small cadre of experts there is no consistency imposed on the way they go about fault diagnosis. As a result, it seems likely that many faults are being mis-diagnosed: attributed to the wrong causes, only partially fixed, or fixed in such a way that new errors are introduced in other parts of the system.

As part of the solution to this operational problem, I created a generic checklist for "Software Fault Diagnosis". This consists of 4 stages (Symptoms, Scenarios, Sources, Solutions), each containing a number of steps. We are now customizing this checklist for the system in question, with the intention of giving it to each developer to put on their wall. It has 2 purposes: as an artefact capturing the knowledge currently in key individuals' heads (which can be maintained along with the system itself), and as a means of improving how everyone does fault fixing.

Interestingly, when I came to draw up the generic checklist, I assumed that it would be possible to find such an item for download somewhere on the Web. Yet a wide trawl of the research literature revealed nothing suitable - certainly nothing applicable to a large-scale, highly distributed system. I had to combine elements from many different sources to put it together.

What has all this to do with BPM?

In my last post I touched on the problems with verification of large processes designed using mainstream BPM applications. Put simply, currently there does not seem to be an equivalent in the BPM world for the various forms of testing deemed essential when software is constructed by any other means. Following this train of thought, there is a next logical question to ask.

TAKE AWAY

Suppose you have just introduced a large-scale process, built using a BPM suite, and the process is not working as expected. What tools do you have for finding the fault?

I will be very interested to hear how readers of this blog are going about it. From my own observations, I suspect that fault diagnosis in BPM-based systems is currently very immature. This may not be surprising when you consider that, even after over 50 years of commercial software, fault diagnosis in conventional software is not yet standardized - as evinced by my inability to find a standard procedure for it in the research literature.

However, at least with conventional software, there are well-known and proven techniques (such as binary search) and tools (such as interactive debuggers) for fault fixing. For fault diagnosis, as for testing, it seems that BPM is not yet delivering enterprise-strength solutions. This represents a serious business risk of which any current or prospective BPM user needs at least to be aware.

Posted by keithhb in Business Process Management | Permalink | Comments (1)

Abraham Maslow, creator of the hierarchy of human needs, famously said: "If the only tool you have is a hammer, you tend to see every problem as a nail." But what if you have a Swiss Army Knife?

This is the 12th and final article in my blog series on The Future of Programming. The series started back in March, and has addressed various topics including:

• Dealing with acronym hell


• BPM without a BPMS


• Managing outsourced application development


• Professional development for developers


• Quality planning


• Managing change in large-scale development

In this concluding article, I will discuss a theme that underlies all these topics.

When I entered the IT industry, I was trained to become not just a software developer, but a software engineer. What is the difference?

Engineering is fundamentally about techniques and tools. An engineering job consists essentially of assessing the situation, deciding what techniques and tools are suitable, then applying them. Hence, in order to be an engineer, you need not only depth of knowledge, but also breadth - a wide-ranging understanding of problem types and solution types.

I began my IT career in the late 1980s. At this time, object-oriented programming was just getting started. Some would say that it is still is! However, there were certainly fewer mainstream programming languages back then, fewer (and less powerful) development tools, and far fewer programmers. Also there were hardly any packaged applications for business use. As a result, it took a lot longer and cost a lot more to introduce new software into an organization.

As a result, it was more common for large-scale software projects to be conducted on quite a formal basis. Not only was the choice of tools and techniques more limited, making it easier to apply an engineering approach to software development, but also the stakes were higher. Despite this, many projects failed, of course, but at least there was recognition - fueled by the writings of thinkers such as Fred Brooks - that software development needed very careful handling if it was to succeed.

Now, in today's world, we have a plethora of new development tools - including BPM suites claiming to make it possible for business users to build their own enterprise software, and SOA suites claiming to make it possible for an organization of any size to share bits and pieces of code willy-nilly. As a result, many organizations are dispensing with the costly and time-consuming engineering processes, in favour of what appear to be the latest and greatest quick-fix solutions.

Well, why not, you may ask? I have 2 main fears about current development trends.

First, organizations seem to be confusing tactical and strategic solutions. A set of BPMN flowcharts may speed up your logistics process, or shave 1% from your invoicing costs. But this is no substitute for an organizational model, a model that is based on high-level business goals. It is easy to adopt short term point solutions that in the medium term fragment your enterprise IT architecture, and in the long term render it unmanageable. In particular, the true problems of SOA Governance are only just starting to emerge. BPM and SOA may in fact be driving a return to the integration nightmare that they promise to solve.

Second, whatever happened to testing? I recently heard a BPM Suite vendor speak proudly of a BPMN process they had implemented that contained 250,000 steps. This represents a vastly complicated piece of business software, so I asked how they had tested it. He replied that testing was unnecessary since their tools did not require the user to write lines of code (just to draw diagrams), and anyway their products contained simulation features if you really wanted to prove that an application worked as expected. He then went on to say how this approach must be OK, since other organizations are using their suite to build safety-critical applications.

This response made me blanch, and still sends shivers down my spine. What sort of world is being constructed using such tools? Whether created via diagrams or as text, software needs testing - and simulation is no replacement for structured, automated, exhaustive, regressive verification. Further, anyone who knows anything about the theory and practice of simulation will recognize immediately that simulation of a flowchart of 250,000 steps is itself a hugely complex and risky exercise.

TAKE AWAY

No matter what tools you use, it is highly dangerous to take the engineering out of software development - yet this is exactly what is happening with the emergence of new BPM and SOA tools.

If you are responsible for software development, I suggest you ask yourself a simple question. Do I care what happens 12 months down the line?

If not, then you can probably ignore much of what I have written in this series.

Otherwise, I suggest you stop and take stock. Remember that much of what you read online and in magazines is written by people with a vested interest in getting you to adopt certain new commercial products. It takes only common sense to see that the practice of software development is nothing more and nothing less than engineering - and engineers in other fields are very cautious indeed about using new techniques and tools to build big things. Why should software be any different?

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

This is the eleventh in a blog series on the Future of Programming. Last time I started to explain the diagram of a "collaborative process pattern" that helps to bring large software projects under safe management control, by structuring the way you handle the continual revision to the development process that inevitably arises when different people are responsible for different aspects of the design. Please refer back to previous articles in this series for details. This time I will finish discussing (for now) how the diagram can help you to manage large software developments, by explaining the third and final perspective from which to understand it - the solution point of view - and how to put the diagram into practice as a means of managing ongoing operations.

From a solution perspective, the Role Activity Diagram is divided into two collaborative transactions - one at overall design level, and one at domain design level. Each transaction is bounded above and below by a 3-way interaction. In each case, the interaction at the top represents an exchange of requirements while the interaction below represents an exchange of solutions.

Both the requirements exchanged in the top interaction and the solutions exchanged in the bottom interaction will involve complex, compound objects. A solution interaction, for example, may pass across:

• The final problem statement, now fully developed
• The objects that form the solution, contributed by different Roles
• The models and tests that justify the solution
• A map of the dependencies between all these objects (for use in tracking the impact of changes) and a related approval chart, showing who has approved what. (Both these can be maintained automatically. Whenever a participating Role submits a new version of one of their deliverables, all the approvals that are dependent on the new deliverable are removed and new approval activities are triggered. Processes and object relationships interact - as long as they are both represented.)

In between the upper and lower interactions, each participating Role will carry out various different activities, in order to provide its own part of the solution. The Role Activity Diagram shown here is simplified and omits these activities, as well as the phases into which the activities will be broken during the sub-project. Why?

All we need at this stage is to represent a consensus from the initial meeting on how to proceed with the sub-project. We are not seeking to prescribe exactly what everyone will do to meet the requirements placed on them - just to establish a common ground on how the issue as a whole will be dealt with. What we are interested in initially is to make it clear to all parties what their responsibilities are to the process as a whole. We need to show only who is to do what, how they will deliver their outputs, and what impact each person's work has on the rest of the process.  The diagram effectively models not only the work required, but also the executive and management controls required over the sub-project.

Once the work gets going, it is inevitable that changes will occur to the process shown in the Role Activity Diagram. In particular, the activities in the sub-project will be extended, and broken into distinct phases, and during the course of the sub-project the elements of the diagram will be extended further as required. The project participants will make agreement depictions at overview or detailed levels that not only refine the process, but also allow all concerned to share the consensus reached about next steps.  In particular, the deliverables may change - perhaps certain ones will be altered to include only partial functionality, and the deficit remedied by other means (or just agreed with the customer as a concession against requirements).

It is here that day-to-day management control can be applied, in order to ensure that the process as it unfolds in operation takes place safely, in accordance with any applicable regulations, and with all parties committed to the same manner of proceeding.

Moreover, it will be possible to do this at either level without contravening the executive control applied from above, since it is clear from the diagram how this control is applied - via the application of the REACT pattern. Should the process participants at any time decide that this application was inappropriate, this change can be described in a simple way via a new Role Activity Diagram, and submitted to the appropriate Role for approval.  For instance, suppose that in a particular domain - safety engineering, say - that the Domain Design Authority feels it sensible to share some of his or her Research activity with the Designers of their part of the solution. After discussion with the Domain Sponsor and Designer(s) in question, the Role Activity Diagram can be altered to show the proposed process change, and sent to the Domain Sponsor for official approval.

There is a lot more one can say about such changes, but for now all we need to do is note that the diagram provides a simple means of achieving shared understanding on a new way forward, and of ensuring that this way forward is safely managed.

TAKE AWAY

Perhaps the primary concern for anyone charged with a large-scale software development should be management of faults, issues and other things that in some way force change to the current plan.  It is a law of nature that such changes happen during software development.  With small-scale software development, agile techniques suffice.  With large-scale software development, my experience, both first- and second-hand, is that they don't.  Having separate teams and/or sub-projects operating concurrently changes the situation fundamentally, and you need something more - something industrial strength - in order to cope.

In the last few articles in this series I have outlined a technique that can help with large-scale software development.  In the next - and concluding - article in this series, I will offer some observations on general emerging trends in enterprise software development.  In particular, I will discuss how the adoption of new BPM/SOA tools is bringing with it some extremely dangerous practices.  To find out what they are, and how to avoid them, tune in next time.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

To understand the pattern fully, it is necessary to interpret the diagram from three different perspectives:

1. Management
2. Participant
3. Solution

Let's start with the management perspective. This is based on the Human Interaction Management (HIM) principle known as separation of control.  Part of this concept is to divide management responsibilities into executive control (setting goals and responsibilities) and management control (monitoring and supporting the work itself).  Executive control comes from above - management control comes from within.  We also need the HIM concept of REACT, an acronym describing the breakdown of any work activity into 5 stages:

1. Research - find out what you need about the domain;
2. Evaluate - apply this information to the work in hand;
3. Analyze - decide from the options available;
4. Constrain - plan the work;
5. Task - do the work.

There is a lot more to separation of control and REACT than suggested here, but that will do for now.  Applying these ideas to the diagram:

• At the overall level, the Overall Sponsor is the Sponsor Role embodying executive control, while the Domain Sponsor and Overall Design Authority together embody management control. The Overall Design Authority has the Research, Evaluate and Analyze activities, while the Domain Sponsor has the Constrain activity. The Task work is not shown as a single activity, but split between Domain Sponsor and Overall Design Authority, and consists of work assignment and work validation respectively.
• At the domain level, the Domain Sponsor is the Sponsor Role embodying executive control, while the Domain Manager and Domain Design Authority together embody management control. The Domain Design Authority has the Research, Evaluate and Analyze activities, while the Domain Manager has the Constrain activity. As above, the Task work is not shown as a single activity. In this case it is split between all three Roles Domain Manager, Designer and Domain Design Authority, and consists of work assignment, design work itself, and validation of that design work respectively.

As the process unfolds, the Role Activity Diagram will be refined to include sub-components of each high-level activity depicted at the initial stage.  This can be done without contravening the controls established at the start - the pattern is fractal, i.e., it can be applied at a more and more detailed level without change to its structure.

The second way to interpret the diagram is from a participant's perspective. There are three Roles at the overall design level, and another three at the level of each specific design domain or discipline (database design, user interface design, security engineering, and so on). There is a degree of correspondence between these levels that indicates that the model is natural:

Overall design level:

1. Overall Sponsor: Acknowledges that the problem is real and kicks off the solution process
2. Domain Sponsor: At the request of the Overall Sponsor, initiates work in each of the design domains, coordinates the separate streams of work, and reports on progress
3. Overall Design Authority: Responsible for the integrity and quality of the whole design (this Role gives permission for the integration of the specific solution)

Domain design level:

1. Domain Manager: At the request of the Domain Sponsor, assigns expert resource to the problem as it applies to a specific domain, then facilitates and supports this work
2. Designer: Participates in the collaborative activity to develop a domain-specific component of the solution
3. Domain Design Authority: Responsible for the domain-specific aspects of the integrity and quality of the design (this Role gives permission for the domain-specific components of the solution to be released for integration into the overall design)

Different Roles, such as 1 and 3 at each level, may well be performed by the same person. However, since Role Activity Diagrams do not include Users, this is not shown explicitly. What we gain from the Role Activity Diagram with respect to process participants is an overall context for each person's work. This is done in a simple and effective way by the Role Activity Diagram, from which it is immediately obvious how each person's work relates to that of the others, who communicates with whom, and who is dependent on whom.

For instance, it is clear that a domain solution cannot be submitted until each component of that solution has not only been prepared, but has also been "Declared Complete" by the corresponding Domain Design Authority. Further, it is not even possible to submit a domain solution component that has not been approved. Submission can only be done in tandem with the Domain Design Authority, as part of a single interaction that also involves approval - a review meeting, perhaps.

TAKE AWAY

Complex problems - such as large-scale software development - do not have to have complex solutions.  The way forward is to apply abstraction: remove distracting detail, and generalize the situation, so as to arrive at clear and simple patterns underpinning all such situations.

The pattern shown in the Role Activity Diagram is such an abstraction.  In HIM terms, it is a collaborative transaction.

Next time I will explain the third and final way to read the Role Activity Diagram - from a solution perspective - and show how to get started with applying these principles in order to safely manage the development of your own software solutions.  No matter how large and complex your development process appears to be, there is always a simple way to manage it safely.  You just need to look at things from the right angle.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

In the last posting to this blog, I wrote that most change management procedures as documented in a Quality Plan give "absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design."

To deal properly with concurrent design, one must somehow support the unpredictable impact across the entire project of design changes originating from one particular team.  In other words, one must stop being scared of change - trying to prevent it, deny it, or limit it.   Shift happens, and success means accepting that change is a natural part of the software development process.  One must institutionalize continual decision-making on what to do next.

This will break down badly if such decisions are imposed from above, since engineering is a highly complex discipline.  Put simply, decisions that do not involve those working at the coal face will usually be wrong.  So all concerned need to share in the decision-making process.

Agile methods have their root in this understanding.  However, talk to enough software project managers and you will find that agile projects tend to run into trouble when they are over a certain size.  It is hard to define the threshold precisely, but from experience I would put it at somewhere around 30 developers.  Why?

Here is a quote from Martin Fowler, generally accepted as one of the key authorities on Agile methods, taken from his seminal article on Continuous Integration

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

Here is the source of the problem in a nutshell.  Most agile practices assume that the mainline is shared between all developers.  This works fine for small projects, but is totally inappropriate for larger projects, where concurrent design is employed and as a result the work is effectively structured as a set of inter-related sub-projects.  Such sub-projects may diverge widely at times - some may never make it into the mainline at all, while others may only ever contribute part of their own output to the final deliverables.

A good comparison is with school class sizes.  Teachers tend to struggle when class size grows beyond 30.  Well, so do software project managers.  In both cases, the solution is to divide people into groups that are semi-independent, and can work at their own speed, making their own choice of subject matter and how far to pursue it.

In my view, there are 2 reasons why the software development community has not properly taken this on board:

1. Failure to understand configuration management techniques.  This is not really excusable, given that there are simple guidelines that anyone can follow to make use of multiple concurrent branches entirely safe.  Laura Wingerd of Perforce Software gives a great explanation of how to manage what she calls the Flow of Change: How Software Evolves (book excerpt) and presentation on same material.  The techniques she explains can be used with any modern software code repository, such as CVS or Subversion.
2. Failure to manage continual decision-making on what to do next.  This is more excusable, since patterns for such collaborative work have only recently emerged, in particular with the discipline of Human Interaction Management.

TAKE AWAY

Are you responsible for (or just working on) a large software project?  If so, you need to think seriously about the consequences of concurrent design.  Here is what IBM has to say (Best practices for software development projects, 10 August 2006):

Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion.

It makes no difference what you are building, or how - whether you are automating business processes using Eclipse MDA, automating business processes via a BPM Suite, or developing a stock trading system.  The IBM article above goes on to list as the first practice crucial to success:

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Exactly.  However, you won't deal properly with the continual changes arising from concurrent design by trying to impose a rigid set of procedures.  What you need are organizational patterns naturally suited to a dynamic, evolving work environment.  In the next instalments of this series I will say more about such patterns, and how to implement them.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (1)

My last entry generated enquiries by email.  I mentioned that "In my own consultancy work, I provide clients proposing to engage in an outsourcing project with a learning trail that their key technical staff should follow, customized to the particular type of project they are considering" and some people asked for examples.  So here is such a learning trail, for Eclipse Plug-in Development using Java.

Note that such a trail can be used both by developers employed by the outsourcing supplier and by those people in your own organization with technical responsibility for the projects concerned.  Doing so will ensure that all the people involved have a proper understanding of the technologies and approaches required.  This is the first step in establishing the kind of close and mutually supportive working relationship that is necessary if the projects are to succeed.

The second step is to decide what outputs you expect from each project.  This will be a lot more than a set of software code modules, since you also need everything necessary for maintaining the solution(s).  Most projects take care to produce the various forms of application documentation required by new users, regular users, power users, system administrators, support staff, operations staff, and so on.  They may also have a requirements document and/or some form of system specification.  However, there are some additional outputs that often get paid only lip service, yet are crucial to success:

• System Architecture.  While software documentation is increasingly being integrated into the code itself, as well as into the static/dynamic models used to generate the code, some high-level information should always be kept separate, and its ongoing maintenance treated as a priority.  This information includes a description of the system's purpose and how it achieves it, the decisions made about key implementation issues such as concurrent use, deployment considerations, licensing strategy, placeholders for future enhancement, and so on.
• Technical plan.  It is necessary to consider up-front the techniques to be used for developing the software, and choose suitable tools with which to implement them.  Such techniques range from high-level work management practices, for example the many variants of agile methodologies, to low-level code development matters such as configuration management strategy.
• Quality plan.  How will you ensure that (every release of) the application meets user needs?  Typically, a combination of strategies is necessary, ranging from code-level practices such as test-driven design and continuous integration through to quality assurance practices based on the principles of Human Interaction Management.

Let's note some things about these documents.

First, such documents are required whether you are implementing business processes via a BPMS, implementing business processes via Model-Driven Architecture, or implementing anything else at all.  Whether you are building a stock trading system for a bank or an embedded device driver for a mobile phone, software construction has certain general principles that must be adhered to.  The rise of agile thinking, and the associated rise of "New Methodologies" designed to handle requirements changes on the fly, have led many people to abandon tried-and-tested software construction practices as old-fashioned.  Beware of throwing the baby out with the bathwater!

Second, the documents do not have to be long.  Rather, the shorter the better.  As long as they cover the required ground, and are consistent with your actual working practice, 3 pages are always preferable to 30.

Third, if you are not going to keep these high-level documents up-to-date, you may as well not bother creating them at all.  Anyone who has ever developed any software at all knows that it always seems to be a moving target.  Everything changes under your feet from the very first day.  In the white heat of a development project, the extra effort required to maintain high-level documents such as those described above may seem like the last thing you should be worrying about.  But actually it is almost the most important thing you can do.

TAKE AWAY

The high-level software project documents described above are  not only your personal bomb-proofing - the evidence, if things go wrong, that you personally acted as a responsible professional - but also the bomb-proofing for the project as a whole.  Armed with current, complete high-level documentation it is possible to salvage many a software project that appears doomed - certainly such documents make it far easier to hand the work over to a new outsourcing supplier, or take it back in house.  Without them, you are working blind.

My first step when engaged to rescue an ailing software project is to make sure that the documents described above are in place and up-to-date - creating them from scratch if necessary, as a retrospective exercise.  Before you can make decisions about where to go next, you need to know where you are now.

In the next postings to this blog I will say more about how to guarantee the success of a software development - and show, in particular, how the often-overlooked Quality Plan can be used to make life easier for everyone.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

• Analysts have a lucrative business in providing feature comparisons, advising vendors which new functions the market would be most interested in, and helping organizations select from the bewildering array of products on offer;
• Consultants have built a niche helping organizations choose, implement, and maintain these beasts - what a French consulting company with whom I once worked referred to disparagingly as "pousse-bouton" work;
• Enterprise architects charged with business process implementation are able to divest themselves of responsibility by laying it on the shoulders of software product vendors and their promises;
• Software product vendors have found a new market, often for a mixed bag of legacy or bought-in products that they can assemble, revamp, and re-purpose as a BPMS suite.

Nevertheless, it can't last forever.  The major vendors in IT are already owning up to the imminent death of the BPMS implicitly, not only by incorporating BPM functionality into monolithic middleware suites (Oracle, SAP, IBM, etc) but even into the operating system itself (Microsoft).  As I pointed out in my last post, published research from IBM shows how to implement business processes without a BPMS at all.  And here is what David Frankel of SAP has to say about using Model-Driven Architecture (MDA) for business process implementation on bptrends.com this month:

Model-Driven Service Orchestration

Often times, developers trying to get a handle on MDA approach it with the thought that they have
to use it to generate the components of their applications, or at least to generate fat skeletons of
the components, to which programmers add some finishing code. Then they run straight into the
last mile struggles with legacy back end systems that hold so much of the data in non-normalized
form.

Some of the most successful approaches that I’ve seen take a different route to nailing down the
last mile with MDA. They use service-oriented architecture (SOA) to present existing application
functionality as a set of service components whose interfaces are highly normalized via
consistent use of service interface languages such as Web Services Description Language
(WSDL). They use modeling to define specific orchestrations of the services. They walk the last
mile by compiling or executing the orchestration models.

In sum, this approach deals with the last mile problem that non-normalized back end systems
pose, by using SOA to provide a normalized intermediate tier that model compilers and model
execution engines can target in a consistent fashion.

In other words, Frankel is saying that you can use MDA to execute mechanistic business processes via SOA.  Draw your business process models, using whatever notation you like as long as it can generate code - UML is fine, for instance.   Then connect your code to services via WSDL.  That's it.  You can use a BPMS if you wish.  But you don't need to.

TAKE AWAY

Some people are attached to BPMS tools because of the wide range of features they typically offer - for messaging, reporting, configuration, and so on. However, tempting as such goodies may be, their use is not in any way related to increased business success.

In fact, there is a strong argument that adoption of multi-functional workplace tools has exactly the opposite effect.  A wide range of features and options often does little but distract people and muddy the waters.  Being bombarded by information from every side and in every form imaginable is a primary cause of stress - when you add multiple forms of messaging to multiple forms of reporting you get what I call "network overload", which is even worse than information overload.  On top of this many organizations are now using sophisticated workplace tools to transfer system administration responsibilities that properly belong to the IT department - authentication and authorization, for example - to end-users who are neither adequately qualified nor adequately resourced to handle such tasks.

What I see is that the net effect of introducing ever more highly powerful applications into the workplace has simply been to obscure individuals' true business goals and responsibilities.  How many people feel that they are running to stand still these days?  Or even just trying to keep their feet on the ground in the face of a tidal wave of information from all sides - and the demands for response generated by such information?

What the IT industry should be delivering is agile tools - agile in the sense of just-good-enough.  On this basis, the rise of programming techniques based on Model-Driven Architecture will, for many organizations, make a BPMS unnecessary.

For a start, an MDA approach is far closer to the original Third Wave ideal for BPM than the current generation of BPMS tools, since mainstream MDA tools such as Eclipse EMF are based on a simple, general, robust underpinning - unlike any current BPMS products!  It could be argued that the OMG MetaObject Facility that underpins EMF is the closest we currently have to the sort of generic framework envisaged by Smith and Fingar in their seminal 2003 book.

More importantly, MDA makes it possible to define your business goals, then implement processes to support them as quickly and simply as possible - and go round the loop again each time the goals change, as they do continually.  So why do you need anything else?  KISS, as they say.

Happy Easter.

Posted by keithhb in Business Process Management • Management • Office Applications • Security • Service-Orientated Architecture | Permalink | Comments (8)

And it's not just me saying that - it's IBM.

Last week's posting to this blog explained why a first step in building a service layer in your enterprise is to ensure that you grasp the general shape of a modern development infrastructure. In other words, you must ensure that your organization possesses a mature and up-to-date software development capability. This is not as simple as it sounds, not only because there are a huge range of competing technologies out there, but also because vendors of middleware systems are doing their very best to spread FUD (Fear, Uncertainty and Doubt) in the marketplace. Haven't got a BPMS yet? Not doing BAM? No enterprise Business Rules Management System? Tch tch tch.

In fact, many BPM practitioners - and certainly most BPM thought leaders - agree that BPM is a management discipline, not a set of software technologies. And this view goes not only for pure BPM, but for supporting applications like BAM, BRM, CEP, and so on. Technology is simply an enabler of management practice. Get your practice right, and you can often do all this kind of thing very cheaply and easily without need for any big iron middleware at all.

Long-time readers of this blog will remember that, as a simple illustration of this, I often mention the number of monthly downloads of JBoss application server - generally over 200,000 - and point out how this by any reckoning this must dwarf the total number of BPMS installations ever performed, anywhere, at any time. Last year I spoke at the largest BPM conference in Europe and the largest programming conference in Europe. Attendance at the former was in the hundreds. Attendance at the latter was in the thousands.

So why does low-level programming still rule the roost - is it about price? Hardly. Only recently has the success of the open source movement made low-cost or free development tooling a reality for enterprise-level software construction. Further, there are now various free and open source BPMS products available. What it's about is functionality.

Taking the Java world as an example, all the advantages claimed for applications such as a BPMS are now available out of the box for plain old Java programmers. There are low-level technologies that automate process implementation from models - see the IBM white paper mentioned below, for example. Business Activity Monitoring, aka BAM, can be handled very well using aspect-oriented tools such as AspectJ in combination with a dynamic configuration framework such as Spring. Business rules can be implemented, and changed on the fly, using a combination of the Strategy design pattern and flexible deployment technologies such as JMX. These are only examples - there are many more ways to build everything you need for software support of BPM using mainstream programming tools that (if you wish) are available for free and as open source.

BPMS vendors and those with a vested interest in the adoption of such systems (for instance, vendors of associated technologies such as business rules) will no doubt respond to this assertion with the rallying cry: "You may be the kind of techie who just likes programming, but our systems place control in the hands of business people, permitting true business agility".

Oh, come on. As mentioned in the last post, you will always need IT people to manage the configuration and deployment of business processes. No business person is ever going to be able to change anything in a BPMS or associated system - and if they have permission to do so, it ought to be taken away!

What a business person does have every right to expect is for it to made abundantly clear to them, in a plain manner suitable for a non-technical stakeholder, what their systems do. Then there is a proper basis for discussion between them and their IT staff on how the system can be configured to behave differently in future. Then their IT staff can go away and implement the requirements of the business.

Over the last 15 years, I have seen far too many cases in which an enterprise application vendor's promises of tech-free administration, together with a sexy user interface, led to quite disastrous mis-configuration by end-users. Some of these cases would make anyone blanch. And once you have been bitten by the business consequences of such unfortunate meddling by people who have not been trained to understand the consequences of their actions, you will be quite happy to leave the IT department in control of back-end systems forevermore. Business people, give the techies your requirements and let them "make it so"! It's their job, not yours, and everyone will end up happier.

TAKE AWAY

I think it is inevitable that BPM systems and programming technology will converge - and that this convergence will be driven by the programming community, not by the BPM community.

For a start, best-of-breed programming tools now possess a degree of smarts to which no BPM system even comes close. This is partly thanks to the emergence of software design patterns during the early 1990s, and the consequent rise of frameworks based on such patterns, but also due to a number of breakthrough innovations in the last decade such as model-driven development, test-first design and aspect-oriented programming. Modern programming tools are the result of years of massive-scale community effort, and incorporate thinking about software that easily surpasses the best efforts of any application vendor.

Further, since you need IT people to make changes to systems, why not let them do it using the tools they know and prefer?

To conclude, here is an extract from a recent IBM white paper:

Once BPM systems are implemented, they are very hard to change because they are engineered as software development solutions that are linear and rigid or because the monitoring solution derives from process models. Solutions derived from processes are flexible but not comprehensive enough to include the nonprocess metrics needed to represent the full state of a business. Thus, a BPM approach not based on models can fall short of fully meeting business needs.

The abstraction of the BPM solution to higher-level models, as we propose, overcomes the shortcomings of BPM alone. It enables business analysts and system architects to contribute directly to the solution. The MDD approach to BPM means that business goals can be defined independent of an information technology (IT) platform. Business-level models either provide linkage to or can be automatically transformed directly to IT-level models using transformation routines. MDD can quickly reflect changing business goals and monitoring needs through models.

Model Driven Development for Business Performance Management, IBM Systems Journal, Volume 45, Number 3, 2006

Like these IBM researchers, I believe that BPM system implementation in future will be an end-to-end programming effort driven by modelling. My own recommendation if you seek a software infrastructure for BPM is to leverage state-of-the-art modelling tools such as Eclipse EMF, together with a low-level code library for the mechanics of state machine implementation (preferably a free and open source solution such as jBPM). This is not only a practical and low-cost route, but also one that satisfies the need among business people for an object-based description of their processes.

So would you like to discover how to prepare your own enterprise for implementation of such solutions? Then stay tuned to this blog.

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (7)

... well, just about everything.

Some of you will atready know that HumanEdj went on general release yesterday.  The press release is here and you can listen to a podcast with Elizabeth Book about it here.

This release follows months of beta testing, in a programme that included over a hundred organizations of all sizes, types, sectors, and geographical locations.  I knew from my consulting experience that the software met a need in all sectors, so the variety in the programme was not really a surprise. What was a pleasant surprise was:

• The level of response, since the only invitations to join the beta programme were mentions on this blog
• How many major incumbent software vendors were on the programme.

When I first started writing about Human Interaction Management (HIM), at the start of 2005, there was considerable resistance to the ideas from the software community.  Despite my efforts to explain that HIM is not competitive but complementary to all existing offerings, raising rather than diminishing the value of current software products, many people involved in the software market seem not to have believed this at all.  Rather, they saw the emergence of a "new breed of productivity software" as a threat.  This viewpoint has definitely changed - out of necessity.

We are currently reaching the peak of a hype curve - not only for existing middleware solutions (Business Process Management, Content Management, Business Rule Management, etc) but for the clutch of good-looking new tools known as Web 2.0 (wikis, blogs, AJAX, etc).  How much money are organizations actually making from adoption of any of this?  Very little, I would say.  And saying it, I often feel like the little boy in the story about the Emperor's new clothes - it can't be long before others start to say it too.

When the wave breaks, as it will soon, incumbents providing these products and services will either go under (i.e, watch consumers become disenchanted with the offerings into which so much money has been invested) or surf - by providing their user base with a step-change in how they extract value from such offerings.

Human nature being what it is, it will not be long before the people currently investing in new middleware start to ask how it differentiates them from their competitors - and those still in love with Web 2.0 start to reject reading blogs, editing wikis, and making customized charts in a browser as a waste of time.  To keep these customers, you must offer something more, something that speaks directly to their most immediate need - which is to succeed in whatever it is that they are currently trying to accomplish.

The business person's true needs are not for more information on a Web portal or more features in an ERP suite.  Rather, the business person needs to become more productive, both personally and on behalf of the employer on whose success they depend.  Hence the business person desperately needs a new type of software tools - tools that will help them achieve rather than just "do stuff".

TAKE AWAY

The coming change is actually an opportunity, not a threat, for a supplier of Web portals and ERP suites.  By providing goal-directed collaboration tools that are customized to integrate seamlessly with their existing offerings, such a supplier can show people how to leverage such offerings for direct and immediate advantage, thus increasing both adoption of these offerings and customer satisfaction.

This is why so many software vendors signed up to beta test HumanEdj.  It is a free goal-directed collaboration tool designed from the ground up to support such integration - it can be branded and extended via standard Eclipse plug-ins.  What software vendor wouldn't think it a good idea to offer something based on this to their customers, something that in helping them meet their own business goals incidentally brings them back to the vendor's own fold?

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications • Service-Orientated Architecture | Permalink | Comments (0)

I have been arguing that current approaches to SOA are fundamentally flawed.  The fundamental problem is that they all treat business activities as a set of tasks organized into a flowchart.  In reality, organizational life is both more complex and simpler than this.  It is more complex because: work of all kinds is more like a set of interacting objects than a flowchart.  It is simpler because: work of all kinds is more like a set of interacting objects than a flowchart.

This is not just me being a smart-alec (read wise-ass, if you're in the US).  Flowcharts may seem simple, but their poor match to reality makes them inappropriate as a foundation for business process implementation.  Even in routine, repetitive work (what Human Interaction Management calls "mechanistic"), to which flowcharts are most suited, there are always a plethora of exception cases - and Pareto's rule tells us that it is the 20% of exceptions that cause 80% of the costs.  And when it comes to adaptive, innovative, interactive human work (what Human Interaction Management calls "human-driven"), flowcharts do nothing but get in the way.

If SOA is to deliver on its promises, it needs to deal properly not only with the exception cases - the edge cases that inevitably fall out to human handling - but also with the human work that at present is the biggest obstacle to improving organizational efficiency.  Contrary to how it might seem, SOA is in fact an ideal technology tool for improving both of these areas.

In both edge cases and human-driven processes, the work consists of a business process in which participants play Roles, via which they each have access to specific documents and other forms of data.  Crucially, the definition of these Roles does not derive from a set of flowcharts, but on goals and responsibilities.  Each Role has various overall goals and responsibilities, which taken together must satisfy the goals and responsibilities of the process as a whole.  A participant creates documents/data while playing a Role - in each case with certain of those goals and responsibilities in mind.  Participants exchange documents/data via Interactions between their Roles - again, in each case with specific goals and responsibilities in mind.

This is a new way of looking at business processes, leads to new and exciting principles and patterns for process description, and offers all sorts of advantages to the business.  Not only does it scale down, to the lowest-level routine processes, but it scales up, to the highest-level strategy and tactics of the organization as a whole.  At both ends of the scale, services can be used to great advantage by partially or fully automating key parts of the business process. In the next post to this blog, I will conclude this series on SOA by considering how a goal-directed approach to business modelling can be used to tame the Minotaur: truly integrate organizational management with the IT backbone.

TAKE AWAY

The way forward for SOA is to adopt an approach to modelling business activities that:

• Has natural support for human-driven activity
• Starts by defining goals and responsibilities, not a set of tasks to be carried out
• At a high-level, is based on business objects rather than flowcharts
• Allows low-level flowchart-type processes to be integrated into a cohesive model of organizational operations.

One immediate benefit of such an approach is that ROI on SOA initiatives can be maximized.  Not only can services be developed to support a larger group of business processes - i.e., human-driven processes as well as mechanistic processes - but the support thus provided is integrated naturally into the work of the organization as a whole.

The only approach currently on offer to meet these needs is Human Interaction Management, aka HIM.  A recent report by Information Age, Riding the fourth wave, heralded the emergence of HIM as follows:

A new generation of people-centric collaborative information management tools is set to produce the first fundamental advances in personal productivity since the arrival of the spreadsheet.

Gartner are quoted as predicting that the market for HIM systems is "at least another 5 years away".  So, the question is this.  Are you going to wait, and try to catch up with the market in a few years?  Or would you prefer it that the market struggles to catch up with you?

Posted by keithhb in Business Process Management • Management • Office Applications • Service-Orientated Architecture | Permalink | Comments (0)

In the last post, I argued that to gain business advantage from SOA, you need first to step back and consider what the fundamental needs of an organization truly are.  To get started on this, I gave a very generic breakdown of the sorts of people involved in running any organization, and at an extremely high level, their most pressing problems.  These problems fall into 3 broad categories, which can be summarized as follows:

1. Commitment Processing [board level] - how to get the organization to implement the desired aims and constraints, which means gaining commitments to these aims/constraints and then monitoring their progress
2. Process Improvement [managerial] - how to make business processes more efficient, which means imposing enough structure that their progress can be facilitated, monitored and measured
3. Personal Productivity [everyone] - how to make the best use of your working day, which means reducing information and network overload enough to leave time to get on with your work

In fact, all these problems are closely related.  What is the connection?

In each case, the problem has the same foundation - that people lack the means to base their work directly on goals.  They struggle since they are trying to structure work on a completely different basis - that of the tasks they imagine work to be comprised of.

Ask any business person - a manager, a salesperson, a designer, a specialist in some field, a business analyst, a consultant, or anything else - to describe a particular type of work carried out in a particular organization, and they will do exactly the same thing.  First they will break the work down into a set of tasks.  Then they will explain how the tasks join up: