This blog entry is the first in a new series. In the next few postings I will be looking at enterprise software development practices in general, with the aim of setting into context various current movements from a management viewpoint. In particular, I will be discussing the relationship of application programming (whose state-of-the-art is evolving rapidly at the moment) to other modern approaches (in particular, SOA and BPM), and showing how the use of all 3 approaches can be properly synchronized within the enterprise.

If you have a say in how software development (by which I include the implementation of SOA services or BPM processes) - is carried out in an organization, or organizations, these next entries will provide you with important guidance. The IT industry is now in a state of ferment similar to that of the 1980s, when object-oriented programming started to gain widespread traction - yet the variety of techniques and tools currently available makes it far harder now than it was then to understand what "best practices" consist of, particularly from the standpoint of a CIO or enterprise architect.

Why is this? In a nutshell, because the choice of software development tools is now bewildering. In the 1980s, a reasonably sophisticated software development tool was Emacs, whose macro support made it a quicker way of creating and testing programs than a vanilla text editor. 2 decades later, people are still using text editors with macros (and some are still using Emacs), but only the diehards work without the support of a vastly more complex "IDE" (Integrated Development Environment). A typical IDE provides templates, offers wizards, generates code/documentation, automates compilation/testing, supports deployment to a range of platforms, synchronizes the work of a team, and so on - a range of features that once you have used them, makes the idea of going back to text editing alone seem rather quaint.

So what's the problem? From the sound of it, everything is hunky dory. Unfortunately not. Many organizations are sinking into a morass of unstructured practices that can only lead to meltdown in the end. Just when everyone thought it was safe to go back in the water after replacing hundreds of User-Developed Applications (e.g., departmental Access databases and Excel spreadsheets) with more or less centralized ERP systems, integrating their legacy mainframe applications via EAI, and fixing the millennium bug, a new danger is on the horizon - what you might call system overload. In the same way that Access and Excel made it possible for departments to spawn their own mini-systems all over the place, so the new generation of powerful software development tools is making it possible for the same people to do it all over again - but this time the products of their effort are vastly more complex, and integrated more tightly into back office automation. They will be a lot harder to harmonize or get rid of than the databases and spreadsheets ever were.

Broadly speaking, there are 3 classes of software development activity going on at the moment in the enterprise:

1. Developing self-contained systems, whether as standalone applications or as customizations to third party packages.


2. Creating building-block components to be exposed as services via SOA tools.


3. Defining end-to-end business processes to be executed via a workflow/BPM engine.

Considering first the programming of self-contained systems, I have discussed in previous entries how by far the bulk of enterprise technology is still implemented via packages (with or without customization) and custom applications. And programmers working on such systems have the benefit of lots of new and powerful techniques to play with, all of which are accessible in a simple way courtesy of their IDEs - pattern-based design, test-first coding, aspect-orientation, model-driven architecture (MDA), inversion-of-control containers, and so on. Yet all these techniques are aimed at single system development - in themselves they provide no means of synchronizing programming work across a large organization, and in some cases (MDA, for example) they actually make it harder.

SOA and BPM vendors may respond to this by claiming that their technique is, in fact, the route to solving just this problem - "buy our toolset and all your integration problems will go away". However, examine the writing of experts in either field and you will be recommended to start small (even if you think big). Third wave BPM was presented as something of an inverse to the big bang approach of 1990s BPR, in which companies were urged to implement wholesale radical change, and the use of small-scale pilots in SOA is encouraged by thought leaders. A helicopter view may be advisable, but in the absence of any more formal means to acquire one than the diverse best practice kitbags of numerous consultancies, it is probably safest to dip a toe into the water rather than dive straight in and bet your business on wholesale transformation via SOA or BPM.

The heart of the matter is that computer science has, until recently, made no real inroads into the problems of large-scale human collaboration. When I first started an MSc Computation course at Oxford University, I was handed a pamphlet containing "THE MATHEMATICS OF PROGRAMMING" - An Inaugural Lecture delivered before the University of Oxford on 77 October 1985 by C. A. R. HOARE, Professor of Computation. In this short lecture, Hoare captured something of the academic spirit of the time in his summary of the principles of the Institute of which he had just become head:

• Computers are mathematical machines.


• Computer programs are mathematical expressions.


• A programming language is a mathematical theory.


• Programming is a mathematical activity.

Our present failure to recognize and use mathematics as the basis for a discipline of programming has a number of notorious consequences. They are the same consequences as would result from a similar neglect of mathematics in the drawing of maps, marine navigation, bridge building, air traffic control, or the exploration of space. In the older branches of science and engineering, the relevant physical and mathematical knowledge is embodied in a number of equations, formulae and laws, many of which are simple enough to be taught to children at school. The practising scientist or engineer will be intimately familiar with these laws, and will use them explicitly or even instinctively to find solutions to otherwise intractable problems.

As a result of these concerns, since 1985 we have made more or less progress towards a mathematical understanding of how computer systems operate, and some of the lessons learnt have been applied to business technology. But there is little recognition either inside or outside academia of the need to formalize the principles via which people collaborate to create business systems - and it takes only a smattering of common sense to appreciate that this is just as important.

TAKE AWAY

Consider the various streams of human activity concerned with package installation/customization, system development, SOA or BPM within the organization(s) you are concerned with.

• Are these streams of activity carried out in a integrated fashion?


• Do all involved know of each other, and interact in a systematic, constructive way to ensure that consistency is maintained?


• Are there effective controls in place to ensure that the different parties are working together to transition business operations towards fulfilment of tactical and strategic organizational goals?

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

In my consultancy work I have seen many situations in which people who knew they could deliver real business value by utilizing open source tools found their efforts strangled by corporate insistence on lengthy evaluation processes. Apart from the time and effort involved in going through such processes, which in itself is a serious deterrent, the cost of carrying out such processes is so high that you might as well buy commercial software from a leading brand.

Consultants and lawyers are the ones who really love this kind of thing - it keeps them in work, and they can make a good case for such an approach to senior executives by stressing risks. In fact it is partly the FUD (Fear, Uncertainty and Doubt) generated in this way that holds back the adoption of open source tools. The criteria typically listed for evaluating open source tools are in fact equally applicable to commercial software, but usually not applied in anything like such force to organizations that have an "Inc" or "plc" in their name - and it's harder to anyway, not only since commercial companies are much less transparent than the open source community, but also because they may have less of an issue with honesty. Large manufacturing companies may think it normal to hire a business integrity consultant to check out a new parts supplier, but few CIOs do so with their software vendors.

As I commented in the first entry in this blog series, the hazards of open source are not those typically those voiced as objections by consultants anyway – unstable or insecure software, availability of support, and legal issues. The open source projects I have discussed, for example, are perfectly viable from all these perspectives. In general, major open source software applications are written at least as well as leading commercial products (often by the same people), enthusiastically supported by expert and helpful developers (as opposed to knowledge-free call center staff), and transparently licensed (via industry-standard agreements). More to the point, anyone with enough experience in IT knows that leading, expensive commercial products are often deeply buggy, poorly supported and legally vulnerable.

So, how should one evaluate an open source system? Is there an easier way to determine the risk?

In my own software development work, I have used many different open source products, and always go through the same 3-stage process when selecting one.

First, I evaluate my own need as clearly as possible. What is it that I am really looking for? People often work under false assumptions when evaluating new software - assuming, for instance, that they need a database when all they really need is a means of saving objects to permanent storage, or that they need Web service tools that support the full WS-* stack when all they really need is a means of enabling communication between distributed components. Conversely, you may be looking for a tool to help write HTML or XML when you would be better served by a higher-level system that generated such low-level code automatically for you, concealing all the complex details.

This step is the fundamental one if you have any interest in open source. The reason so many open source projects exist is that people start them to meet real-world needs, of which there are an infinite and varied number. Somewhere out there, there is probably someone who has had exactly the same problem as you, who has solved it by writing the tools they needed, and who has then decided they may as well make them available to others via open source. If you are considering open source at all, you may well find you can get exactly what you need, not only what is offered by a limited number of commercial vendors.

So let's assume you have applied such lateral thinking and found a set of tools that meet your needs as closely as possible. Which ones are safe for enterprise adoption?

The second step is to tick some basic boxes - those above, for software stability, availability of support and legality. This is the work of minutes, not days. A project with 1 committer who last posted an update 3 years ago, that has no stated users, and whose license consists of "use this as you wish" is not a good bet - but you can feel secure in choosing a project that:

1. Has a number of committers who post regular updates


2. Can demonstrate a user base


3. Is backed by VCs or major companies


4. Uses one of the standard open source licenses (assuming that the conditions of the license do not preclude your intended use of the software).

Having made a shortlist of tools that are both suitable and viable, the third and final step is to ask yourself a single question - and again it is the same question you should ask when evaluating commercial software. What would we do if this project ended? In other words, is it possible either to maintain the open source software in question in-house, or replace it by another product?

Most enterprises would look for the second option, since the current trend in IT is to divest oneself of responsibilities rather than incur new ones. If you are of this mind, here are the things you should bear in mind when answering the question:

• The replacement product does not have to be open source. If you eventually have to switch to a commercial product, you have simply gained some license-free time from initial use of an open source product, which should be more than enough to cover the cost of the migration.


• The replacement product does not have to work in the same way as the original product, or provide the same functionality - it just has to support the usage you intend to make of the open source product. In a simple case, a replacement for your email server only has to support the configuration you have implemented, not every possible configuration available in the product you currently use. For a more complex case, consider JBoss jBPM, for instance, recommended as a free BPM solution in the previous post to this blog. This is packaged as a Java library, that can be used standalone in any Java program or with any application server, can export suitable processes to BPEL, and though the programming model of JBoss jBPM is very powerful (see the last post) it is based on a standard Petri-net paradigm dating back to the early 1960s. So let's suppose the JBoss project ends for some reason, though this is highly unlikely. Existing jBPM processes can be supported for as long as desired in any other Java execution environment, those based on Web service orchestration can be transferred when desired to any BPEL engine, and it is essentially a matter of legwork to port other processes to another workflow system that is likewise based on Petri nets. If you are using the more advanced features of jPDL, you will have to make the odd change here and there, but in the end this is a completely conventional trade-off between functionality and portability - a trade-off that enterprises make every day when utilizing software products, whether open source or commercial.

TAKE AWAY

Open source tools represent a major source of advantage for the enterprise - not just because they are free of license charges (since it is often wise to pay for support anyway), but because in general the open source marketplace offers a range of functionality that commercial vendors struggle to match, hampered as they are by the need to support a legacy product range and provide a consistent offering. All enterprises are well-advised to consider open source when maintaining any aspect of their IT and software development infrastructure, and the good news is that you do not have to pay consultants through the nose to do so - just follow the 3 steps above, and use your own common sense to make practical decisions about the software you adopt.

Posted by keithhb in Open Source | Permalink | Comments (0)

This is the 3rd entry in a blog series on open source - this time I will introduce another major open source project of which every enterprise should be aware, and next time I will discuss a quick and simple way to evaluate open source projects in general.

This time it is the turn of JBoss jBPM. There are a number of open source workflow/BPM projects, ranging from simple BPEL engines to fully-fledged suites. However, the JBoss offering is particularly interesting, for 3 reasons:

1. JBoss is, these days, about as blue-chip as you can get in the open source world. Not only do they offer a complete range of middleware, but they have download rates that must be the envy of other products (typically hundreds of thousands per month) and are venture-capital backed. Implementing JBoss now is like buying IBM used to be.


2. jBPM supports BPEL, but is based on a more powerful programming model - what they call Graph Oriented Programming - with its own process definition language, jPdl. This language was designed from the ground up to support the workflow patterns developed by Prof. Wil van der Aalst and his group.

   Why is this important? Because the workflow patterns attempt to represent all possible modes of behaviour in a "programmatic" business process - preset workflows in which human involvement is limited to key decision and data entry points. So if your process definition language can represent all the patterns, you know you can automate just about any such business behaviour using it. As far as I know, jBPM is the only workflow system that claims to support all the patterns.



3. jBPM is closely tied to the component architecture of the entire JBoss suite. In my recent blog series on BPM Futures, I discussed how process automation is effectively becoming a graphical interface to component tooling. This approach is very much in line with the architecture of jBPM. Web services, for example, can be employed as desired, but there is no need to move program control flow into a weak language such as BPEL unless it is necessary - you can do as much or as little as you want in Java, for instance.

TAKE AWAY

There is a lot of discussion about BPM at the moment - the debate around jBPM is typically lively - and a lot of this debate is still along the lines of: so what is BPM and why do we need it?

As I have commented before in this blog, despite the views presented by analysts and vendors, anyone working at the coalface in a range of companies will know that by far the bulk of enterprise process automation is still done using ERP packages and component technologies rather than using dedicated workflow/BPM tools.

I discussed in previous entries how new approaches may well change that, as "declarative" techniques enable the automated generation and application of certain types of executable process. In future entries, I will be looking in more detail at the implications of this for enterprise architecture. For now, however, anyone seeking an enterprise-strength process automation system would be well-advised to look at jBPM. Not only is it a powerful open source solution, but it fits naturally with emerging techniques and tools (coming with its own Eclipse plugin, for example).

In the next and concluding entry in this blog series on open source, I will discuss how to decide when you should not adopt an open source approach - without needing to spend a fortune on consultancy advice to find out!

Posted by keithhb in Business Process Management • Open Source | Permalink | Comments (2)

This is the second entry in a blog series looking at major open source projects of which every enterprise should be aware. The series will conclude with a discussion of open source adoption issues, providing a quick and simple way to understand when and where to avoid open source.

The subject of this blog entry is the extraordinary software framework known as the Eclipse Platform. Originally developed in-house by IBM, Eclipse was open sourced in 2001 via the creation of a supervisory body known as the Eclipse Foundation, a not-for-profit consortium that now boasts 115 member organizations. IBM still put a lot of money and development time into Eclipse, but for years they have been supported in this effort by some of the largest companies in the world.

So what is Eclipse? In a nutshell, the first software product with the potential to remove Microsoft's monopoly in desktop computing.

IBM set out in the late 90's to create a common software framework to underpin all their Java-based middleware products. Further, the framework had to be extensible via plugin modules. They wanted to establish a standard user interface not only for their own product range, but also for any custom applications or supporting third party tools created to go with products in that range.

To say IBM succeeded would be an understatement. The intellectual effort that went into Eclipse was first-class, with direction from such luminaries as Grady Booch, and taking inspiration from the pattern-based approach to software design that is only now becoming standard practice. There are already many hundreds of plugin modules for Eclipse (open source and otherwise), and the framework is becoming the de facto standard software development environment, not only in the Java world but for other languages too - especially as Eclipse runs on most modern operating systems, in each case with a native look-and-feel. The writing is probably on the wall for competing, non-specialized "IDEs" for software development - even those few (like IntelliJ Idea) that have managed to retain a loyal user base will inevitably find it eroded as the massive vendor support behind Eclipse and the growing number of plugins makes it harder and harder to compete.

So why does this threaten Microsoft? Because of something called the Rich Client Platform.

Eclipse was originally developed as a tools framework - you were supposed to use it to create applications for use by IT folk, for example to write software or configure application servers. It took a surprisingly long time to realize that there was no reason to restrict Eclipse in this way. You can use Eclipse to write any kind of application! And doing so makes a lot of sense. Eclipse has a very well-architected plugin model that means you can leverage the framework to get off the ground very quickly with a new application, simply by building it as an Eclipse plugin. And once you have done so, your own application is immediately compatible with all the hundreds of other Eclipse plugins out there. So with release 3, Eclipse acquired a set of standard features (known as the Rich Client Platform) that make it easy to create any kind of standalone application in this way.

Now let's look back at the history of Microsoft. DOS may have been the company's original means of establishing dominance, but in the last 10 years it has retained it because of something else: applications. In particular, the ubiquity of the Office suite has meant that organizations standardized on Windows in order that their documents would be compatible with those from other sources. But that is only part of the story. For a long time now, it has been almost inconceivable for a software vendor to release a desktop product that did not run on Windows. Hence, by using Windows you were guaranteed that all the software you might want at some time to use would in fact be available to you.

Companies like Google realize this very well, which is why they snapped up the Web-based word processor Writely only shortly after its launch. The more people that use a Web browser as an application platform, the less the importance of choosing Windows for your desktop operating system as opposed to (say) Linux. But whatever the "Web 2.0" evangelists may say, the Web browser is never going to be fully-capable, or even durable, as a platform for building applications. The software development model (AJAX) is:

1. Built on sand - Javascript is a scripting language, that was never designed to take the strain of heavyweight software.


2. Badly architected - not only is AJAX a kludge on top of HTTP right from the start, but it tends to lead to poor design, breaking the Model 2 MVC practices that were finally becoming mainstream. This is changing as better practices start to appear for AJAX, but there is no layering implicit in AJAX itself, so there will always be the temptation to write poor code.


3. Server- and browser-dependent - you cannot run an AJAX application without connecting to the server in question using an appropriately capable browser. What about applications that need to run offline and with more choice of device?

AJAX is unlikely ever to seriously challenge Windows as an application platform - but Eclipse may. Unlike AJAX, Eclipse is founded on an enterprise strength language (Java), based on a soundly architected plugin model, and independent of almost any network and device restrictions. As more and more software vendors port legacy products to Eclipse, and release new products in the form of Eclipse plugins, the importance of choosing Windows for your desktop operating system can only erode, a trend which will be accelerated by the recent emergence of Eclipse sub-projects that provide system-level features such as single sign-on (Higgins) and communications (ECF). The tipping point will probably come when someone releases word processing and spreadsheet Eclipse plugins that can (like Writely) use the emerging open standard for document format as well as the document formats for Microsoft Office.

TAKE AWAY

If you are considering development or maintenance of any desktop software application, whether for in-house use or supply to others, look hard at Eclipse before deciding on a framework. It may well be sensible to build the application as, or port it to, a set of Eclipse plugins. Just so that you know I practice what I preach, we are building our own next generation toolset for collaborative work using Eclipse - and have found it to be very productive, especially the embedded facilities for generating both business logic and diagramming code.

In fact, the same consideration should be given even to server-side applications. Eclipse can run "headless" - without an interactive user interface - and by developing your application as Eclipse plugins you gain access to all the rich functionality of the framework and its existing plugins (many of which are open source). For some reason, the Eclipse Foundation to date have not highlighted this capability of Eclipse, but no doubt they will realize in due course what an opportunity they have been missing and cater to it via a "Rich Server Platform" feature.

Finally (and this is where we came in), the rise of Eclipse has serious implications for any enterprise considering future desktop strategy. There may be less obstacles than you thought to replacing your desktop operating systems with (for example) a Linux variant. Look at what is available now in Eclipse, what is coming, and consider again how much money you will really need to spend on Microsoft licenses in the next few years. Such considerations have a direct impact not only on the desktop machines and licenses you purchase, but on your development strategy - the need for .NET capability along with J2EE may be less than you thought, for example.

Tune in again to this blog for a discussion of another major open source project with the potential to change the enterprise computing landscape. In the next entry I will pick up again on the discussion in the last blog series, BPM Futures, and show how open source tools can be used to implement a forward-thinking BPM strategy.

Posted by keithhb in Open Source • Operating Systems | Permalink | Comments (10)

In the next few blog entries I will be discussing open source projects that have reached a certain stage – that of being at least as mature as their commercial competitors. Many organizations are still reluctant to use open source software for mission-critical applications – and in some cases there are good reasons to be suspicious. I will conclude this series of blog entries with a discussion of the potential hazards of adopting certain types of open source software in an enterprise infrastructure.

However, these hazards are not those typically those voiced as objections by IT management – unstable or insecure software, availability of support, and legal issues. The open source projects I will be discussing in the next few entries are perfectly viable from all these perspectives. In general, major open source software applications are written at least as well as leading commercial products (often by the same people), enthusiastically supported by expert and helpful developers (as opposed to knowledge-free call center staff), and transparently licensed (via industry-standard agreements).

Similarly, the advantages of open source are not those typically quoted, either. For example, open source evangelists, being technical folk, make a big deal out of being able to correct or enhance the code yourself if you need to. This is exactly the opposite of what an enterprise is looking for - the very last thing they want is the headache of untangling some huge and complex application when there is a problem. The real benefit of open source is that it is generally more attuned to real-world needs, since open source projects get started precisely in order to meet such needs. While software vendors are grappling with legacy products and market positioning, the open source community just goes off and builds useful stuff.

Stay tuned to this blog for a discussion of the true hazards of open source as regards enterprise adoption, which are to do with the type of project. For now, I will be looking at projects that are not only free from such hazards, but that offer significant advantages over their commercial rivals.

First up is an offering in a space of increasing importance – information retrieval. The success of Google is a measure of how valuable this technique has become in modern life, yet despite the continuing advance of Web search, the facilities that most organizations offer for searching their intranet, or even their public-facing Web site, are little more than pitiful. The ability to retrieve a particular document from the sprawling Web presence (internal or external) of a large company is often more a matter of art than science, and may well depend on knowing in advance the likely path to the data concerned.

This is, of course, a well-known problem, to which knowledge management techniques are often touted as the answer. Indeed, such advanced solutions can be employed to unlock the information archives of a company – but you do not always need the sort of high-end, and very expensive, tools sold for this purpose. For one thing, there are alternative approaches to knowledge management that can be leveraged to expose the knowledge hidden inside information, some of which I discussed in previous blog entries, and I will be returning to this topic in future posts. However, there are also far simpler ways by which the hundreds of thousands of documents available via HTTP on a large company’s servers can be made available – and thus turned into an asset rather than a liability.

I was struck recently by the marketing puff for a commercial search engine, which offers as a “step forward” the ability not only to search structured data (requesting specific values for specific fields) but also to provide flexibility via assigning weights to the different terms in a search. Surprisingly, whoever wrote this PR spiel – and possibly the vendor itself – does not seem to be aware that such “advanced” features have been available for years in the leading open source search engine.

The search engine in question is Lucene, an Apache project. Lucene is not only very well-established but can do some very cool things. For example, Lucene can search via named fields - a feature that in itself offers a step on the way to full knowledge management – and offers wildcard, fuzzy, proximity and range searches. Terms in any of these types of search can be boosted, grouped, and controlled via the use of logical operators.

For a proper explanation of these features, see the links referenced above. The point is that Lucene is very full-featured. However, Lucene is not a search application – it is simply a search engine, a code library that can be used to interrogate any body of text. To use Lucene as the search tool on a Web site, for example, it must be embedded into a product designed for the purpose.

Fortunately, since June 2005 there has been a sub project of Lucene aimed at doing precisely this. The Web crawler and search facility that incorporates Lucene is Nutch. Nutch operates somewhat like Google - in fact Nutch incorporates some technology that Google itself put into the public domain, technology that permits Nutch to crawl, index and search enormously large collections of documents. Moreover, the user interface of the Nutch search facility is, if anything, more helpful than that provided by Google, providing an analysis of how the page ranking was generated along with the results themselves.

If I were a search engine vendor, I would be worried now that Nutch is getting off the ground properly. I have used Nutch in anger, and cannot see any reason to buy commercial software for this purpose now. Simple to install, robust, scales, configurable - what more could one want?

TAKE AWAY

If you work for a large organization, ask yourself whether its public Web site and intranet provide genuinely acceptable search facilities. If not, consider implementing Nutch. It takes – literally – minutes to do the initial installation, and configuration even for a large document base is not complex. And Nutch, like Lucene, comes from the Apache Foundation, one of the most (if not the most) well-established and reliable homes for open source projects.

Tune in next time for a discussion of other open source software with the potential to transform your enterprise IT environment for only a small investment of time and effort.

Posted by keithhb in Internet • Knowledge Management • Open Source | Permalink | Comments (2)

This is the concluding entry in a 5-part series on BPM futures. Previous blog entries showed that:

1. The imminent provision by the OMG of a standard XML format for BPMN will effectively render BPEL unnecessary
2. The adoption of BPMN for process definition means that business process automation will become a high-level graphical interface to component technologies, rather than a separate layer in enterprise architecture
3. By taking a computer science perspective, we can see that in due course Web service choreography will supersede Web service orchestration for the definition of those processes in which human involvement is limited to key decision and data entry points (what I call "mechanistic" processes)
4. Hence, forward-looking organizations need to consider now the creation of choreography descriptions, an activity for which the techniques of Human Interaction Management can be used: both to develop the processes needed to establish agreement on multi-party collaborative contracts, and to provide the graphical notation needed to depict the contracts themselves.

In this final blog entry (for now) on BPM futures, I will look at the type of software vendors best positioned to take business process automation forwards.

The argument above shows that business process automation is turning into a rather different sort of activity than has been supposed to date. As implemented by current BPM suites, business process automation is largely a matter of drawing flowcharts - effectively it is a sort of high-level, graphical programming. In fact, BPM competes directly with a combination of the UML and J2EE, for example. Though the picture painted by BPM vendors, and some analysts, is rather different, the battle could be considered a David vs Goliath one. I would happily bet that the total number of all BPM installations ever is nowhere even close to the average number of downloads of JBoss (say) in a single month - which in November 2005 was about 200K.

Of course, a BPM suite provides features that do not always come for free with a component-based approach to process automation - in particular, Business Activity Monitoring (BAM) - though here too technologies such as JMX are starting to provide a viable alternative. However, my point is simply that to date BPM has not become an ubiquitous part of enterprise technology, and this is because it is competing with other approaches that are mature and very well-established. Moreover, the people who are in the end responsible for implementing and maintaining BPM - the IT department and their outsourcing suppliers - are a lot more comfortable with J2EE or .NET than they are with a particular proprietary BPM toolset.

However, all this may change with the next generation of BPM tools. The argument that was made in previous blog entries, and that is outlined above, shows that business process automation will inevitably move towards the creation of multi-party collaborative contracts - where the parties may be internal to a single organization or span several organizations - and the use of these contracts to generate executable business processes. This is a very different kettle of fish from the usual software development process. Though agile methodologies, in particular, place an emphasis on negotiation and re-negotiation of client expectations, the future focus of BPM will be on the creation of agreements among several collaborating parties. BPM tools will then provide the means for each party to automatically turn their part in such agreements into working software.

Tools of this nature bear little relation to existing BPM products. But they do bear a strong relation to another class of products - those provided by vendors specializing in middleware to support Service-Orientated Architecture (SOA). SOA techniques are in flux at the moment, and there is a lot of debate among experts as to the benefits of a distributed vs centralized backbone, or the WS-* stack over simpler techniques such as REST and XML-over-HTTP. However, despite all this debate, there are some tenets to which most people adhere - in particular, the key technique of SOA is to establish what are generally called service contracts. There is little agreement among those in the field even on what a service is, but most agree that a service should expose some kind of description as to what it does. A service contract is a description of itself that a service publishes (somehow) - it is used by those that wish to call the service to determine what they can expect from it.

There is a strong parallel here to the direction in which business process automation is going. Both are based on the definition of contracts rather than on the definition of internal behaviour. SOA tools have not yet advanced to the point of being able to generate the latter from the former, but (as shown in the previous blog entry) there is every reason to suppose that this can and will be automated by software in due course.

There is a growing realization that the architectural principles emerging for the design of business processes by analysts (based on establishment of multi-party service contracts for collaborative behaviour) are the same as those already used by technicians in SOA. In fact, there is a strong argument that in a decentralized, globalized, Internet-enabled economy it will soon be hard to survive without applying such principles to the organization of your enterprise. As McKinsey wrote recently:

"the shift from transactional to tacit interactions requires companies to think differently about how to improve performance - and about their technology investments. Moreover, the rise of tacit occupations opens up the possibility that companies can again create capabilities and advantages that rivals can't easily duplicate."

Hence, it is realistic to expect that the next generation of tools for business process automation will come neither from platform vendors, nor from BPM pure-plays, but from vendors currently providing a complex, and to many people invisible, part of the IT middleware infrastructure - tools that come under such confusing and often overlapping categories as Web Services Management (WSM), Enterprise Service Bus (ESB), Enterprise System Management (ESM), and so on.

TAKE AWAY

In the next year or two, expect SOA to become high-profile among business leaders as well as technical folk - to become as much a management methodology as an approach to IT. This change will bring with it some key questions, related not only to development - how can we create a set of services, and agree on them with those responsible - but also to governance - what policies, procedures and other controls do we have in place for maintaining and regulating services? The SOA community is evolving some answers to these questions, but inevitably these answers are currently too IT-focused. If SOA is to successfully make the transition into a management approach, it needs to take on board the general principles and patterns that underpin business activity.

Such principles and patterns need not only to encompass diverse fields of business theory (quality management, knowledge management, project management, ...) but also insights from the many scientific disciplines connected with organizational life (psychology, biology, social systems theory, learning theory, and so on). Further, it is no good assembling such insights into a huge, diverse and confusing set of "best practices" - the consultant's favourite trick. They must be organized properly into a theory with a formal basis that provides the reassurance of sound thinking together with a structured and maintainable approach to implementation.

With respect to IT industry support for this move, expect SOA vendors to find an exciting new field opening up for them - the full support of mechanistic business processes, currently considered the domain of BPM tools. Those SOA vendors that embrace the move to declarative expression of processes descriptions, and automatic generation of process implementations, may well find themselves catapulted to a new and prominent level of visibility among business leaders.

PS

On a personal note, before publishing the ideas in this series I did not expect them to be met with much (or any) agreement. BPEL in particular is a focus of current IT industry interest, and consequent vendor investment. So it was a pleasant surprise not only to find that some influential people have already come to similar conclusions about BPEL - Phil Gilbert of Lombardi, for example, in his own blog - but also to receive many messages of support for the ideas in general. Whether you agree or disagree with the views expressed in the series, please let me know, either publicly via a blog comment or privately via email (see my bio). Private messages will of course be treated in confidence.

In the next blog entries, I will be looking at some of the most important open source software tools. I am continually surprised in my consulting work to find that organizations are paying for software that is less mature, less feature-rich and less well supported than that provided by the open source community. I will also discuss some of the situations in which open source is not a good idea.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Service-Orientated Architecture | Permalink | Comments (1)