This is the second in a series of 3 predictions, of ideas that I believe will be publicly accepted by the end of 2008. Last time I discussed how "The new wave of IT solutions is far too complex for most people". This time I'll go into some more detail.

In particular, my second prediction - that people will come to see how "The new wave of IT solutions raises as many problems as it solves" - is based on a phrase that 10 years ago one used to see a lot more than nowadays, namely Total Cost of Ownership (TCO).

The term TCO came into widespread popular usage for IT purposes around the mid-1990s, when Oracle Corporation and Sun Microsystems in particular started propounding the advantages of the "Network Computer" - a true thin client that possessed no hard disk, only an Internet connection. However, it took a long time for the advance of the Web to bring with it software that can support thin client solutions without reducing the user experience unacceptably - AJAX-based applications, for example. Even now, nearly all computing devices still possess a hard disk, and in my view will probably do so for the forseeable future.

So vendors no longer bandy about TCO as much as they did, since they see less mileage to be gained by doing so. Instead, IT customers are starting to take on these concerns. Faced with proposals from vendors, and analysis on sites such as ebizQ, that suggest the modern enterprise needs not just BPM, but BPM+SOA+BRM+CEP+BAM+BI+Mashup+add_your_new_type_of_product_here, organizations are starting to worry very seriously about administrative overheads. Which is not surprising, since:

• These products are new.


• They are complex.


• There are a lot of them.


• Hardly anyone understands how to use them properly.

TAKE AWAY

Part of the reason that corporate decision-makers are deciding to focus on the management aspects of process-orientation (rather than the technology aspects) is due to the rise of Business Process Outsourcing (BPO). Why implement what you may not end up owning?

The outsourcing industry has matured enormously in recent years. Talking to outsourcing vendors while in India recently, I was deeply impressed by their realistic grasp of what works and what doesn't in outsourcing, and how customers can get the best from low-cost commodity labour.

Further, the outsourcing model has spread to the point where it is no longer simply a matter of cost reduction (if it ever was). The rise of "homeshoring" attests to this.

However, the increased interest in outsourcing raises questions of its own. Next time I will address my third prediction - that people will recognize during 2008 how "There are fundamental business problems that need help from IT but that are not addressed at all by the new wave of IT solutions" - and discuss where the market will turn for answers to these questions.

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (2)

Happy New Year. As as traditional at this time of year, I will make a few predictions for the IT landscape in 2008.

Or rather, I will predict some acknowledgements - state some ideas that I believe will be publicly accepted by the end of 2008. These are about what might be called the "new wave of IT": the host of new techniques and technologies that have recently become contenders for a place on the enterprise backbone. SOA and BPMS are obvious candidates for inclusion, along with associated acronyms such as CEP, BRM, mashup, and so on.

So here are the acknowledgements that I believe will happen in 2008:

1. The new wave of IT solutions is far too complex for most people


2. The new wave of IT solutions raises as many problems as it solves


3. There are fundamental business problems that need help from IT but that are not addressed at all by the new wave of IT solutions

Over the next few days I'll take these in turn, starting with:

The new wave of IT solutions is far too complex for most people

If there is a single subtext to the various discussions I have with business people, it is confusion. And this is hardly surprising. Consider BPM, for example, and let's look again at Ismael Ghalimi's original definition of "BPM 2.0" from Feb 2006:

BPM 2.0 is not for non-technical business analysts. Never should have been, never will, and nobody should care.
http://itredux.com/blog/2006/02/01/bpm-20/

Despite this frank admission from a founding figure in the BPM space, and the efforts of certain commentators to explain that BPM is first and foremost a management approach, analysts and vendors continue to present BPM as a set of tools aimed at business users. Presumably this is in the hope that the take-up of such tools will thereby be helped, by broadening the potential market. However, if anything take-up has been harmed by this approach - most business people I talk to are apprehensive about BPM tools, even if they are drawn to BPM as a means of organizational transformation.

Let's face it head-on: BPM tools are simply a new form of programming, and as with any other new form of programming, it is not yet well understood how to use them safely. In particular, mainstream programming aids such as automated testing and static analysis are not even available for BPM tools.

It is no different with SOA, CEP, BRM, mashup, and so on. In the end, these are all techie gadgets, and the first step towards making good use of them is to recognize that.

TAKE AWAY

The SOA analyst firm ZapThink, in their own predictions for 2008, write:

many organizations are still struggling with the business case for SOA, or even worse, have made a wrong turn or reached some kind of impasse with their SOA initiatives
http://www.ebizq.net/news/8772.html

It's hardly surprising. There are new business ideas emerging originally from IT, and resulting new software tools aimed at business users, but the larger incumbent software vendors are still busy trying to sell technologies. As a result, business users are either suspicious or struggling (or both) - and in 2008 we may well see a backlash.

Tune in to the next post for some more home truths about the new wave of IT. Welcome or not, you should consider these ideas before trusting your career, and your organization's livelihood, to the new wave of IT :-)

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (0)

In my last post I promised to discuss mashups - in particular, how there is more than one type of mashup.

Let's start at the beginning - what is a mashup?

To help answer this, here are some pictures taken from publicly available presentations by the CTOs of 2 firms supplying software commonly used to create mashups, Coach Wei of NexaWeb and John Crupi of JackBe.

Coach Wei gives an overview of where a mashup sits with respect to SOA and Web 2.0, terming it "Enterprise Web 2.0":

Coach's principle is that a mashup is how you leverage back-end services, namely by applying a rich Web 2.0 interface in order to expose them for business use. John Crupi provides 2 pictures that explain this how this differs from previous approaches, which typically use portal technology. Here we see what John calls "Today's Content-Driven Web":

Here, by contrast, we see what John calls "Tomorrow's Data-Driven Web":

Essentially, a mashup moves the integration of disparate forms of information behind the firewall. By doing so, it becomes possible to add additional controls, not just for security purposes but also in order to connect data more usefully - for example, by showing on a single chart information that previously was displayed in separate "portlets" (i.e., separate areas of a Web page).

The growth of NexaWeb and JackBe testifies to the utility of mashup technology. However, the tools available today are just the start of the mashup trend. Why?

Useful as mashups are in their current form, software applications built in this way are essentially competing with process support systems as service consumers. Like mashups, both the BPMS (for routine, semi-automated work) and the HIMS (for human collaborative work) use services to implement business processes.

In practice, a current mashup can be thought of as equivalent to a step in a business process - a node in a BPMN diagram, for instance, or a Task in a HumanEdj Role. In fact, it would be quite possible to design a BPMS or HIMS process that incorporated one or more mashups in this way. However, at this point mashup and process tools are not integrated. For now at least, there is a clear division in the market between mashup creation tool vendors and process support system vendors, and this can only lead to problems.

Most organizations have come to accept that managing work activity better means managing work processes better. Hence they are using process techniques, and in due course most will adopt process execution software. If they are also using mashups, the consequence is that they will then have 2 siloed forms of design activity for "service-oriented business applications". As a result, features belonging really to processes will creep into mashups, and vice-versa.

In particular, processes of either BPMS or HIMS type tend to use data items to preserve their "state" - i.e., where the work has currently got to. For example:

• A BPMS process representing a product assembly might contain data items corresponding to quality check results, and use these to determine whether or not the product is ready to ship.


• A HIMS process representing a marketing campaign might contain data items corresponding to focus group results, and use these to determine whether or not the collateral is ready for release.

TAKE AWAY

Like the Web, mashups will evolve. The next stage is surely to provide a means of integrating Rich Internet Applications with BPMS and/or HIMS processes, to permit safe sharing of data between applications and processes.

Such integration is likely to start with the HIMS rather than with the BPMS, since the processes that they deal with are very different. Processes for which a BPMS is suited are routine, repetitive and semi- or fully-automated. Human involvement in such processes is limited to key points, and takes the form of low-level data entry and decision-making. This is not the territory of mashups, which are aimed usually at higher-level knowledge work activities. Mashup activities are much more closely related to HIMS processes - innovative, collaborative, adaptive human work. It is HIMS processes that can leverage the rich content of a mashup, and hence it is here that the integration will probably start.

Returning to the theme of this post, there are essentially 2 types of mashup: siloed and process-aware. For now, mashup tools are siloed. However, in due course they will become process-aware via integration with HIMS technology - and once this happens, the combination of HIMS processes and mashup applications will be an extremely powerful way to leverage both Web 2.0 and your legacy infrastructure.

Posted by keithhb in Service-Orientated Architecture | Permalink | Comments (2)

Just back from SOA India 2007, where I gave the keynote.  The conference was a fascinating experience.  In particular, it was interesting to talk to delegates from many different sizes and types of organization who are actually doing SOA!  Not just talking about it :-)

India really is where it is all happening, IT-wise.

And the discussions I had over the week, both with delegates and with other speakers, bear witness to an urgent need for a deeper understanding of SOA governance.  Many people working in the field are technically savvy, and have a good understanding of SOA techniques and tools, but still possess little or no appreciation of governance issues.  I heard some startling things ...

In the end, the sole justification for SOA is change.  In particular, SOA should enable you to align business change with system change - if you don't expect to attempt this, there is little or no point doing SOA.  However, change in an SOA environment is deeply risky - analogous to rewiring an F16 in mid-flight.  Scary stuff, considering that standard software QA techniques such as static analysis and automated testing have yet to mature for systems that are assembled rather than programmed.

Hence, if you're going to do SOA, you can expect to be doing on-the-fly transformation - and in the absence of standard, systematic control techniques you need to have a very good understanding of how to control change manually.  In other words, you need to define your governance processes, and manage their implementation,  very carefully indeed.  Such processes are not routine, and are only partial candidates for automation - they are, in fact, exactly the kind of processes dealt with by Human Interaction Management (HIM) (collaborative human work processes).

However, no-one I spoke to last week seemed to understand the need for process control in SOA governance, however sophisticated their ideas on SOA in general. Certainly none of the vendors in the governance space have yet incorporated HIM techniques into their tools. The next step for SOA governance tools is to integrate the functions of a Human Interaction Management System (HIMS) such as HumanEdj into an existing, repository-based offering - and the first entrant in this space will have the market to themselves.

TAKE AWAY

I suspect that understanding of the need for process control in SOA governance will only become widespread once there have been a few public disasters - and these cannot be far off.  Soon, early adopters of SOA will attempt to use their brand new system environment in the way promised by its original advocates - namely, to make rapid changes to align IT with new business requirements.  And without effective process controls, reconfiguration of a hugely complex network of interacting services is bound to bring problems.

I see no reason why the impact of such problems should be limited.  Rather, they are quite likely to break some organizations entirely.  It may be that for now, SOA change problems have been covered up - but large-scale operational or financial disaster is not so easy to hide.

So why wait until it happens to you?  Would it not be better to bring some order to the chaos?  In other words, take responsibility for your SOA!  Do not treat SOA governance as if you were going out to dinner with colleagues - each choosing a few items from the menu on a whim.  Rather, treat SOA governance as if you were all working together in the kitchen - collaborating with just enough structure to ensure efficiency, and adjusting your work patterns from day to day as necessary to meet requirements.

You may be interested in my slides from the keynote, which addressed these issues.  Also, following discussions at the conference around data management, I have prepared a new version of my Balanced Scorecard for SOA Governance.  The organizers, SDA India, videotaped a long interview with me which will be online in due course - when I know the link, I'll post it to this blog.

In my next post, I will discuss a related topic - namely, mashups, of which there are more than one kind.  Stay tuned.

Posted by keithhb in Service-Orientated Architecture | Permalink | Comments (0)

Most cathedrals were built before the basic principles of physical dynamics were worked out.  It was not until the 18th century that mathematical physicists such as Poisson and Cauchy developed the ideas that enabled structural engineers to predict stress patterns in a large body.  Until then, architects - or master masons, as they were known - relied on knowledge handed down orally, from one guild member to another.  This knowledge amounted effectively to rules of thumb, albeit highly sophisticated ones.

Hence, even the supreme achievements of mediaeval architecture were constructed without a true knowledge of what actually held the structure together.  The trial and error of centuries had established that certain approaches were valid, and that these approaches scaled well.  This was enough to construct masterpieces such as Chartres.  But they didn't really know why it worked.

Recognizing this, the master masons took steps to ensure the safety of their structures: they employed a technique that modern engineers call "similar redundancy".  Basically, they built more columns, buttresses, arches and so on than they probably needed.  Similar redundancy works well if you are confident in the validity of your approach.

Skipping over the next few centuries to the modern day, engineers in fields that demand high system integrity (such as aerospace, for example) now prefer a technique called "dissimilar redundancy".  This technique involves first identifying the separate functions of a system (raise the ailerons, decrease the pressure, turn on lighting, etc) - then commissioning a solution for each function from several different manufacturers, who are not permitted to discuss the matter amongst themselves at all.  In this way, it is hoped that any fundamental error in one design will not be propagated into the other designs.  All the designs are used together in the production system, so that each function has a main operating solution plus a number of "dissimilar" backup solutions.  Dissimilar redundancy is a stronger technique than similar redundancy.

Turning to SOA, what is the relevance of this discussion?  In an enterprise computing environment pre-SOA, there are generally a number of competing "sources of truth" - for example, different ERP systems, front-end operational systems, data marts, data warehouses, and so on.  It is common to shield senior management from this complexity via a layer of reconciliation activity inside the organization, that attempts to understand the differences, or at least choose a set of figures for presentation to the board.  Nevertheless, one can safely say that most people working on business systems - and certainly all accountants - accept that a "single source of truth" is a complete myth in enterprise life.

With SOA, this starts to change.  The underlying systems are generally still there, of course, but the reconciliation is more likely to be done by IT staff than by managers and accountants, as part of the design of an SOA infrastructure.  Which service do we use to present "total revenue per region per quarter" on a dashboard?  Service XYZ123, from system XYZ.  Or the mean result of XYZ123 and service PQR678 from system PQR.  Or the median result of 10 services ... Or ...

Whatever solution is chosen, the people who really understand what to do about reconciliation (namely, managers and accountants) are out of the loop in daily working practice, even if they were involved in the initial design.  Effectively, dissimilar redundancy has been dispensed with, since the new solution treats the various sources of truth not as alternative implementations, but as part of a single implementation.

Further, even similar redundancy is often dispensed with!  It is quite possible to build an SOA infrastructure in which key services have no parallel implementation (running on different servers, using different middleware, etc).  This is not good practice, of course, but in a large enterprise with thousands of services, it can easily happen - and if it does, a single bug in such a service could bring down your entire operation like a house of cards.

TAKE AWAY

Next week I will be delving deeper into these issues as part of my keynote speech at SOA India 2007, and how to solve them via governance techniques based on Human Interaction Management.  I will also be running a workshop on the last day of the conference (you have to pay separately for this, but apparently, over 150 people have already signed up). If you're at the conference, come up and say hi.

You might also like to have a listen to the podcast that Elizabeth Book (Editor-in-Chief of ebizQ) and I just recorded, in which we discuss the latest version of the free HumanEdj, that provides software support for Human Interaction Management principles.  Elizabeth writes: "I am actually about to download this thing to use in my own office".  Why not follow her example?  And I am always interested in your feedback - do let me know what you think.

Posted by keithhb in Service-Orientated Architecture | Permalink | Comments (0)

With only a couple of weeks to go until SOA India 2007, I've been putting thoughts together for my keynote speech. Something I may or may not mention (haven't decided yet) is a comparison between "architecture" of the service-oriented kind, and "architecture" as practised by the cathedral-building stonemasons of the middle ages. Cathedrals were effectively the mediaeval equivalent of middleware - the large-scale infrastructure via which stakeholders in the Christian religion (i.e., church officials) serviced their customers (i.e., the Christian community).

Looking back so far, it is hard to know exactly what a master mason did when working on one of these huge buildings. However, we have some idea of the principles they used, and some idea also of the working practices employed by the guilds of the time. Interestingly with regard to a comparison with SOA, we also know that most of the work would effectively have been modification rather than construction from scratch. For a start, many of the major cathedrals were constructed over a period longer than a single working lifetime. Further, early cathedral building often proceeded on a trial-and-error basis - for example, there are records of buttresses being built only when certain parts of the structure started falling down.

This somewhat experimental approach changed in the gothic period, when new design principles meant that a small and elegant set of supports could hold up a huge edifice - and moreover, do so while letting in huge amounts of light. This transformed not only the construction process, but the experience of those inside the building - from being huddled in a cold and gloomy environment surrounded by yards of thick stone, to being bathed in golden rays of warming and uplifting sunshine.

So am I saying that SOA is like gothic architecture? A new and vibrant experience for the users, thanks to a step change in technological sophistication?

It may be, but that's not the point I'm trying to make. Rather, my point is about the difference between the two forms of architecture.

In general, cathedrals change their structure, very slowly, if at all. When they do change, it is usually by accretion - the addition of new buildings, rather than modification to the old ones. This means that the original design principles are largely unaffected by change. Hence, if a cathedral is still standing a few years after construction, it will probably still be standing a few hundred years after construction (barring the usual acts of god, or more often, acts of man).

With SOA, on the other hand, we can expect constant and often dramatic change both to the structure and functions of a system. After all, the main USP of SOA is that it helps facilitate such change! This means that original design principles are little or no guide to the longevity of the system. In other words, however clever the original designers were, the whole system can be compromised hopelessly by their successors at any time.

TAKE AWAY

A capacity to deal with change is often touted as the main attraction of an SOA approach. Yet this pre-supposes that change is managed properly. I see no evidence for this happening. Change on an enterprise scale, that spans multiple applications, is a complex human-driven process involving many different parties: usually many different departments, and often many different organizations altogether. The only tools SOA vendors are providing to deal with such processes are repositories for technical artefacts, that sometimes include the ability to make issue lists. In a nutshell, this isn't even close to enough, since it provides no mechanism for Human Interaction Management.

Further, the problem is exacerbated by subtle issues of system robustness - which we can glean from a deeper comparison between cathedrals and enterprise applications. Tune in to my next post for more mediaeval illumination :-)

Posted by keithhb in Service-Orientated Architecture | Permalink | Comments (0)

Abraham Maslow, creator of the hierarchy of human needs, famously said: "If the only tool you have is a hammer, you tend to see every problem as a nail." But what if you have a Swiss Army Knife?

This is the 12th and final article in my blog series on The Future of Programming. The series started back in March, and has addressed various topics including:

• Dealing with acronym hell


• BPM without a BPMS


• Managing outsourced application development


• Professional development for developers


• Quality planning


• Managing change in large-scale development

In this concluding article, I will discuss a theme that underlies all these topics.

When I entered the IT industry, I was trained to become not just a software developer, but a software engineer. What is the difference?

Engineering is fundamentally about techniques and tools. An engineering job consists essentially of assessing the situation, deciding what techniques and tools are suitable, then applying them. Hence, in order to be an engineer, you need not only depth of knowledge, but also breadth - a wide-ranging understanding of problem types and solution types.

I began my IT career in the late 1980s. At this time, object-oriented programming was just getting started. Some would say that it is still is! However, there were certainly fewer mainstream programming languages back then, fewer (and less powerful) development tools, and far fewer programmers. Also there were hardly any packaged applications for business use. As a result, it took a lot longer and cost a lot more to introduce new software into an organization.

As a result, it was more common for large-scale software projects to be conducted on quite a formal basis. Not only was the choice of tools and techniques more limited, making it easier to apply an engineering approach to software development, but also the stakes were higher. Despite this, many projects failed, of course, but at least there was recognition - fueled by the writings of thinkers such as Fred Brooks - that software development needed very careful handling if it was to succeed.

Now, in today's world, we have a plethora of new development tools - including BPM suites claiming to make it possible for business users to build their own enterprise software, and SOA suites claiming to make it possible for an organization of any size to share bits and pieces of code willy-nilly. As a result, many organizations are dispensing with the costly and time-consuming engineering processes, in favour of what appear to be the latest and greatest quick-fix solutions.

Well, why not, you may ask? I have 2 main fears about current development trends.

First, organizations seem to be confusing tactical and strategic solutions. A set of BPMN flowcharts may speed up your logistics process, or shave 1% from your invoicing costs. But this is no substitute for an organizational model, a model that is based on high-level business goals. It is easy to adopt short term point solutions that in the medium term fragment your enterprise IT architecture, and in the long term render it unmanageable. In particular, the true problems of SOA Governance are only just starting to emerge. BPM and SOA may in fact be driving a return to the integration nightmare that they promise to solve.

Second, whatever happened to testing? I recently heard a BPM Suite vendor speak proudly of a BPMN process they had implemented that contained 250,000 steps. This represents a vastly complicated piece of business software, so I asked how they had tested it. He replied that testing was unnecessary since their tools did not require the user to write lines of code (just to draw diagrams), and anyway their products contained simulation features if you really wanted to prove that an application worked as expected. He then went on to say how this approach must be OK, since other organizations are using their suite to build safety-critical applications.

This response made me blanch, and still sends shivers down my spine. What sort of world is being constructed using such tools? Whether created via diagrams or as text, software needs testing - and simulation is no replacement for structured, automated, exhaustive, regressive verification. Further, anyone who knows anything about the theory and practice of simulation will recognize immediately that simulation of a flowchart of 250,000 steps is itself a hugely complex and risky exercise.

TAKE AWAY

No matter what tools you use, it is highly dangerous to take the engineering out of software development - yet this is exactly what is happening with the emergence of new BPM and SOA tools.

If you are responsible for software development, I suggest you ask yourself a simple question. Do I care what happens 12 months down the line?

If not, then you can probably ignore much of what I have written in this series.

Otherwise, I suggest you stop and take stock. Remember that much of what you read online and in magazines is written by people with a vested interest in getting you to adopt certain new commercial products. It takes only common sense to see that the practice of software development is nothing more and nothing less than engineering - and engineers in other fields are very cautious indeed about using new techniques and tools to build big things. Why should software be any different?

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

This is the eleventh in a blog series on the Future of Programming. Last time I started to explain the diagram of a "collaborative process pattern" that helps to bring large software projects under safe management control, by structuring the way you handle the continual revision to the development process that inevitably arises when different people are responsible for different aspects of the design. Please refer back to previous articles in this series for details. This time I will finish discussing (for now) how the diagram can help you to manage large software developments, by explaining the third and final perspective from which to understand it - the solution point of view - and how to put the diagram into practice as a means of managing ongoing operations.

From a solution perspective, the Role Activity Diagram is divided into two collaborative transactions - one at overall design level, and one at domain design level. Each transaction is bounded above and below by a 3-way interaction. In each case, the interaction at the top represents an exchange of requirements while the interaction below represents an exchange of solutions.

Both the requirements exchanged in the top interaction and the solutions exchanged in the bottom interaction will involve complex, compound objects. A solution interaction, for example, may pass across:

• The final problem statement, now fully developed
• The objects that form the solution, contributed by different Roles
• The models and tests that justify the solution
• A map of the dependencies between all these objects (for use in tracking the impact of changes) and a related approval chart, showing who has approved what. (Both these can be maintained automatically. Whenever a participating Role submits a new version of one of their deliverables, all the approvals that are dependent on the new deliverable are removed and new approval activities are triggered. Processes and object relationships interact - as long as they are both represented.)

In between the upper and lower interactions, each participating Role will carry out various different activities, in order to provide its own part of the solution. The Role Activity Diagram shown here is simplified and omits these activities, as well as the phases into which the activities will be broken during the sub-project. Why?

All we need at this stage is to represent a consensus from the initial meeting on how to proceed with the sub-project. We are not seeking to prescribe exactly what everyone will do to meet the requirements placed on them - just to establish a common ground on how the issue as a whole will be dealt with. What we are interested in initially is to make it clear to all parties what their responsibilities are to the process as a whole. We need to show only who is to do what, how they will deliver their outputs, and what impact each person's work has on the rest of the process.  The diagram effectively models not only the work required, but also the executive and management controls required over the sub-project.

Once the work gets going, it is inevitable that changes will occur to the process shown in the Role Activity Diagram. In particular, the activities in the sub-project will be extended, and broken into distinct phases, and during the course of the sub-project the elements of the diagram will be extended further as required. The project participants will make agreement depictions at overview or detailed levels that not only refine the process, but also allow all concerned to share the consensus reached about next steps.  In particular, the deliverables may change - perhaps certain ones will be altered to include only partial functionality, and the deficit remedied by other means (or just agreed with the customer as a concession against requirements).

It is here that day-to-day management control can be applied, in order to ensure that the process as it unfolds in operation takes place safely, in accordance with any applicable regulations, and with all parties committed to the same manner of proceeding.

Moreover, it will be possible to do this at either level without contravening the executive control applied from above, since it is clear from the diagram how this control is applied - via the application of the REACT pattern. Should the process participants at any time decide that this application was inappropriate, this change can be described in a simple way via a new Role Activity Diagram, and submitted to the appropriate Role for approval.  For instance, suppose that in a particular domain - safety engineering, say - that the Domain Design Authority feels it sensible to share some of his or her Research activity with the Designers of their part of the solution. After discussion with the Domain Sponsor and Designer(s) in question, the Role Activity Diagram can be altered to show the proposed process change, and sent to the Domain Sponsor for official approval.

There is a lot more one can say about such changes, but for now all we need to do is note that the diagram provides a simple means of achieving shared understanding on a new way forward, and of ensuring that this way forward is safely managed.

TAKE AWAY

Perhaps the primary concern for anyone charged with a large-scale software development should be management of faults, issues and other things that in some way force change to the current plan.  It is a law of nature that such changes happen during software development.  With small-scale software development, agile techniques suffice.  With large-scale software development, my experience, both first- and second-hand, is that they don't.  Having separate teams and/or sub-projects operating concurrently changes the situation fundamentally, and you need something more - something industrial strength - in order to cope.

In the last few articles in this series I have outlined a technique that can help with large-scale software development.  In the next - and concluding - article in this series, I will offer some observations on general emerging trends in enterprise software development.  In particular, I will discuss how the adoption of new BPM/SOA tools is bringing with it some extremely dangerous practices.  To find out what they are, and how to avoid them, tune in next time.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

To understand the pattern fully, it is necessary to interpret the diagram from three different perspectives:

1. Management
2. Participant
3. Solution

Let's start with the management perspective. This is based on the Human Interaction Management (HIM) principle known as separation of control.  Part of this concept is to divide management responsibilities into executive control (setting goals and responsibilities) and management control (monitoring and supporting the work itself).  Executive control comes from above - management control comes from within.  We also need the HIM concept of REACT, an acronym describing the breakdown of any work activity into 5 stages:

1. Research - find out what you need about the domain;
2. Evaluate - apply this information to the work in hand;
3. Analyze - decide from the options available;
4. Constrain - plan the work;
5. Task - do the work.

There is a lot more to separation of control and REACT than suggested here, but that will do for now.  Applying these ideas to the diagram:

• At the overall level, the Overall Sponsor is the Sponsor Role embodying executive control, while the Domain Sponsor and Overall Design Authority together embody management control. The Overall Design Authority has the Research, Evaluate and Analyze activities, while the Domain Sponsor has the Constrain activity. The Task work is not shown as a single activity, but split between Domain Sponsor and Overall Design Authority, and consists of work assignment and work validation respectively.
• At the domain level, the Domain Sponsor is the Sponsor Role embodying executive control, while the Domain Manager and Domain Design Authority together embody management control. The Domain Design Authority has the Research, Evaluate and Analyze activities, while the Domain Manager has the Constrain activity. As above, the Task work is not shown as a single activity. In this case it is split between all three Roles Domain Manager, Designer and Domain Design Authority, and consists of work assignment, design work itself, and validation of that design work respectively.

As the process unfolds, the Role Activity Diagram will be refined to include sub-components of each high-level activity depicted at the initial stage.  This can be done without contravening the controls established at the start - the pattern is fractal, i.e., it can be applied at a more and more detailed level without change to its structure.

The second way to interpret the diagram is from a participant's perspective. There are three Roles at the overall design level, and another three at the level of each specific design domain or discipline (database design, user interface design, security engineering, and so on). There is a degree of correspondence between these levels that indicates that the model is natural:

Overall design level:

1. Overall Sponsor: Acknowledges that the problem is real and kicks off the solution process
2. Domain Sponsor: At the request of the Overall Sponsor, initiates work in each of the design domains, coordinates the separate streams of work, and reports on progress
3. Overall Design Authority: Responsible for the integrity and quality of the whole design (this Role gives permission for the integration of the specific solution)

Domain design level:

1. Domain Manager: At the request of the Domain Sponsor, assigns expert resource to the problem as it applies to a specific domain, then facilitates and supports this work
2. Designer: Participates in the collaborative activity to develop a domain-specific component of the solution
3. Domain Design Authority: Responsible for the domain-specific aspects of the integrity and quality of the design (this Role gives permission for the domain-specific components of the solution to be released for integration into the overall design)

Different Roles, such as 1 and 3 at each level, may well be performed by the same person. However, since Role Activity Diagrams do not include Users, this is not shown explicitly. What we gain from the Role Activity Diagram with respect to process participants is an overall context for each person's work. This is done in a simple and effective way by the Role Activity Diagram, from which it is immediately obvious how each person's work relates to that of the others, who communicates with whom, and who is dependent on whom.

For instance, it is clear that a domain solution cannot be submitted until each component of that solution has not only been prepared, but has also been "Declared Complete" by the corresponding Domain Design Authority. Further, it is not even possible to submit a domain solution component that has not been approved. Submission can only be done in tandem with the Domain Design Authority, as part of a single interaction that also involves approval - a review meeting, perhaps.

TAKE AWAY

Complex problems - such as large-scale software development - do not have to have complex solutions.  The way forward is to apply abstraction: remove distracting detail, and generalize the situation, so as to arrive at clear and simple patterns underpinning all such situations.

The pattern shown in the Role Activity Diagram is such an abstraction.  In HIM terms, it is a collaborative transaction.

Next time I will explain the third and final way to read the Role Activity Diagram - from a solution perspective - and show how to get started with applying these principles in order to safely manage the development of your own software solutions.  No matter how large and complex your development process appears to be, there is always a simple way to manage it safely.  You just need to look at things from the right angle.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

The trick is to recognize the reality of how people work together - and use that as the basis for a structured approach to managing their activities.

Let's take a typical scenario. A design team is in the middle of a large concurrent design project, and an unexpected issue has arisen - sufficiently serious to warrant specific management attention, and potentially impacting several design areas (database, user interface, security, and so on).

Meeting to discuss the problem, team members realize that a new sub-project is required to manage the issue. Initially they have no plan or structure for this new sub-project. The input to the sub-project is itself a problem definition that may require further investigation and development. However, there are a number of significant things that they can say from the start. They may have none of the detailed elements of a process, but they are able to recognize a pattern. They know how collaborative problem solving generally takes place.

The pattern they recognize for collaborative problem solution has specific aspects that allow us to model it, at a high level, as a process:

1. The people involved will perform certain Roles, each with their own goals and responsibilities. There will be a number of participants in the solution process, with skills in different domains - database design, user interface design, security engineering, and so on. For each type of contributor, there may well be distinct phases of activity: problem definition, concept design, detailed design and testing, integration and testing, and so on. Hence, the sub-project is in fact composed of multiple distinct streams of activity, one for each domain. The contributions of all these have to be coordinated.
2. The eventual deliverables will meet certain preconditions. If successful, the output of the sub-project will be a solution, probably consisting of a set of solution components in different domains. The solution components in each domain will include models and tests that demonstrate that the solution solves the defined problem. There may well be interface specifications and other supporting documentation. Among all these objects there will be varied dependency relationships. Hence, the components of the solution both at domain level and overall are an interdependent set that must be protected from further change unless the whole solution is revisited and retested. The set of solution components can be viewed as a transaction - somewhat like a database transaction, although subject to quite different rules in which all sub-project participants commit jointly to the resolution of the issue.
3. The process participants will interact in a way to ensure that their individual contributions combine to produce overall deliverables of the highest possible quality. The components of each domain solution must be made available to an appropriate person for integration into a unified domain solution. Similarly, the domain solutions must be made available to an appropriate person for integration into a unified overall solution. The individual domain solutions and the overall solution must all be validated and certified by an appropriate authority, and this activity must be linked to the aforementioned integration.

With this analysis in mind, we can use a Role Activity Diagram to represent the sub-project as a whole:

TAKE AWAY

You may be reading this because you are responsible for a large-scale software development. And you may be wondering how the diagram above is going to help you. Is it the magic bullet you need? If so, why?

If you do not find the diagram above self-explanatory, this is because it draws on concepts from Human Interaction Management. Before you can understand the diagram, you need to understand the concepts.

In the next instalment of this series, I will explain 3 different perspectives from which the diagram can be viewed:

1. Management
2. Participant
3. Solution

Armed with this understanding we will start to see how even the thorniest - and scariest - problems in large-scale software development can be tamed. Complex problems can be broken down into a set of simple ones, so that management controls can be applied and the required solutions delivered.

If this sounds like what you need, tune in next time - and if you can't wait, you can always get the book :-)

Posted by keithhb in Management • Security • Service-Orientated Architecture | Permalink | Comments (0)

In the last posting to this blog, I wrote that most change management procedures as documented in a Quality Plan give "absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design."

To deal properly with concurrent design, one must somehow support the unpredictable impact across the entire project of design changes originating from one particular team.  In other words, one must stop being scared of change - trying to prevent it, deny it, or limit it.   Shift happens, and success means accepting that change is a natural part of the software development process.  One must institutionalize continual decision-making on what to do next.

This will break down badly if such decisions are imposed from above, since engineering is a highly complex discipline.  Put simply, decisions that do not involve those working at the coal face will usually be wrong.  So all concerned need to share in the decision-making process.

Agile methods have their root in this understanding.  However, talk to enough software project managers and you will find that agile projects tend to run into trouble when they are over a certain size.  It is hard to define the threshold precisely, but from experience I would put it at somewhere around 30 developers.  Why?

Here is a quote from Martin Fowler, generally accepted as one of the key authorities on Agile methods, taken from his seminal article on Continuous Integration

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

Here is the source of the problem in a nutshell.  Most agile practices assume that the mainline is shared between all developers.  This works fine for small projects, but is totally inappropriate for larger projects, where concurrent design is employed and as a result the work is effectively structured as a set of inter-related sub-projects.  Such sub-projects may diverge widely at times - some may never make it into the mainline at all, while others may only ever contribute part of their own output to the final deliverables.

A good comparison is with school class sizes.  Teachers tend to struggle when class size grows beyond 30.  Well, so do software project managers.  In both cases, the solution is to divide people into groups that are semi-independent, and can work at their own speed, making their own choice of subject matter and how far to pursue it.

In my view, there are 2 reasons why the software development community has not properly taken this on board:

1. Failure to understand configuration management techniques.  This is not really excusable, given that there are simple guidelines that anyone can follow to make use of multiple concurrent branches entirely safe.  Laura Wingerd of Perforce Software gives a great explanation of how to manage what she calls the Flow of Change: How Software Evolves (book excerpt) and presentation on same material.  The techniques she explains can be used with any modern software code repository, such as CVS or Subversion.
2. Failure to manage continual decision-making on what to do next.  This is more excusable, since patterns for such collaborative work have only recently emerged, in particular with the discipline of Human Interaction Management.

TAKE AWAY

Are you responsible for (or just working on) a large software project?  If so, you need to think seriously about the consequences of concurrent design.  Here is what IBM has to say (Best practices for software development projects, 10 August 2006):

Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion.

It makes no difference what you are building, or how - whether you are automating business processes using Eclipse MDA, automating business processes via a BPM Suite, or developing a stock trading system.  The IBM article above goes on to list as the first practice crucial to success:

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Exactly.  However, you won't deal properly with the continual changes arising from concurrent design by trying to impose a rigid set of procedures.  What you need are organizational patterns naturally suited to a dynamic, evolving work environment.  In the next instalments of this series I will say more about such patterns, and how to implement them.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (1)

Here is a typical description of a Quality Plan, selected at random from the Web:

Quality Plan Description

The Quality Plan describes how a developer's overall quality process guidelines will be applied to a project. It defines what is meant by the various quality-related tasks in the Project Plan.

For example, a developer's quality manual may describe a review process for ensuring that delivered software meets requirements. The Quality Plan for the project tailors this general definition to the project at hand, specifying items such as who generates the requirements, what form these will take, who reviews them, etc. Thus, when a task in the Project Plan reads "Review and update requirements," the Quality Plan defines what will be done, and who will be responsible.

Quality Plan Contents

The Quality Plan outlines how you will build quality into the software and documentation. The dates assigned to key tasks in the Quality Plan are entered into the project plan.

The Quality Plan describes:

• How you control changes.
• How you ensure that the product meets the requirements (validation).
• How you ensure that the product works properly (verification).
• How you track multiple development builds of the software to avoid confusion (configuration management).
• How you plan for and execute testing, both incrementally during development and for the entire product before delivery.
• How you track and resolve defects.
• How and when you conduct design reviews, code reviews, walk throughs, reviews of test scripts, reviews of test results (for example, is 100% of all code checked, or only the most complex parts?).
• Definitions, methods, and criteria you use to determine whether the software has passed each review.

At first glance, this seems OK.  However, it is in fact rather a limited approach to quality planning.  The problem with such a definition is that one could get away with almost anything (or nothing) in the Quality Plan document, since the only expectation is to have some form of test and review during the life of the project.  Quality in its classical definition means "fitness for purpose".  So the real question that should be asked of a Quality Plan is this: how does it ensure that the software delivered is fit for its purpose?

A more thoughtful approach to quality planning is that of the PRINCE 2 method developed by UK government, and now an internationally recognized approach.  Here is a typical PRINCE 2 approach to quality planning:

Purpose

The purpose of the Project Quality Plan is to define how the supplier intends to deliver products that meet the customer's quality expectations and the supplier's quality standards:

• Does the plan clearly define ways in which the customer's quality expectations will be met?
• Are the defined ways sufficient to achieve the required quality?
• Are responsibilities for quality defined up to a level that is independent of the project and project manager?
• Does the plan conform to the corporate quality policy?

Composition

The Project Quality Plan should contain:

• Customer quality expectations
• Acceptance criteria, a prioritised list of criteria for the final product(s) that must be met before the customer will accept the final product(s).
• Quality responsibilities, who is responsible for each of the aspect of quality of the final product(s)
• Reference to any standards that need to be met
• The quality-control and audit processes to be applied to project management
• Quality-control and audit process requirements for specialist work
• Change management procedures
• Configuration management plan
• Any tools to be used to ensure quality.

Derivation

• Customers quality expectations and requirements
• Organisational or programme quality management system and standards
• Configuration management and change control requirements

Such a description will produce a more useful (if longer) document.  However, there is still a fundamental problem with it.  The problem lies in the innocuous phrase "Change management procedures".  Typically such procedures comprise some combination of:

• Techniques for issue tracking
• Guidelines for approval of proposed changes.

This gives absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design.

Demands for increased productivity and shorter times-to-market generally force software engineering organizations to adopt concurrent design.  Different parts of a complex application are developed in parallel and then integrated, shortening project schedules. Typical interactions in this type of business process are activities such as agreeing on sub-contract terms, signing off a specification, concluding a project stage, and so on. Each of these may involve several parties, contain processing spread over varied systems on different platforms owned by different organizations, and require a number of steps to complete successfully. Further, although efficiency gains can be achieved via this approach, it carries risks of rework, which arises when concurrently performed work turns out to be incompatible, and an impact of one interaction is that others need to be repeated.

In other words, concurrent design is what Human Interaction Management defines as a typical human-driven process - fraught with instability, since software engineering managers are dealing with activities that are only partly predictable when the process commences. It is a fundamental characteristic of the whole environment that it is organic and dynamic, so hard coded process descriptions are unsuitable. Even if they are supported by tools that make modification of that process possible, the participants are innovative, creative people who need the flexibility to put their skills and experience to use in what seems like the most appropriate way at the time.

There is a parallel situation in the structure of the product itself. A complex software application may appear to decompose neatly into components (and services) but in fact it is a densely tangled set of solutions to a huge and varied set of requirements and constraints. Later modifications only increase the complexity. New connections and dependencies are introduced by tracing the impact of modifications, defining and implementing consequent changes, testing the whole package and restoring the integrity of the product.

TAKE AWAY

Software product managers attempting to deal with applications built using concurrent design generally find themselves tearing their hair out by the time the project is halfway through.  If they are lucky they will be running to stand still.  If not, they will find the project slipping further and further every week.  Failure to properly handle concurrent design is probably the single biggest cause of software project failure.

The solution to managing concurrent design lies in giving the ever-changing nature of a human-driven process the respect it deserves.  In such work, a major part of the workers' duties is to decide on next steps.  This must be done both continually and collaboratively, since there are always new issues to resolve and few of them can be dealt with in isolation.

In the next instalment of this series, I will look at techniques for structuring this "continual decision on next steps".  If you want to make your change management as elegant and efficient as your software code, stay tuned.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (0)

My last entry generated enquiries by email.  I mentioned that "In my own consultancy work, I provide clients proposing to engage in an outsourcing project with a learning trail that their key technical staff should follow, customized to the particular type of project they are considering" and some people asked for examples.  So here is such a learning trail, for Eclipse Plug-in Development using Java.

Note that such a trail can be used both by developers employed by the outsourcing supplier and by those people in your own organization with technical responsibility for the projects concerned.  Doing so will ensure that all the people involved have a proper understanding of the technologies and approaches required.  This is the first step in establishing the kind of close and mutually supportive working relationship that is necessary if the projects are to succeed.

The second step is to decide what outputs you expect from each project.  This will be a lot more than a set of software code modules, since you also need everything necessary for maintaining the solution(s).  Most projects take care to produce the various forms of application documentation required by new users, regular users, power users, system administrators, support staff, operations staff, and so on.  They may also have a requirements document and/or some form of system specification.  However, there are some additional outputs that often get paid only lip service, yet are crucial to success:

• System Architecture.  While software documentation is increasingly being integrated into the code itself, as well as into the static/dynamic models used to generate the code, some high-level information should always be kept separate, and its ongoing maintenance treated as a priority.  This information includes a description of the system's purpose and how it achieves it, the decisions made about key implementation issues such as concurrent use, deployment considerations, licensing strategy, placeholders for future enhancement, and so on.
• Technical plan.  It is necessary to consider up-front the techniques to be used for developing the software, and choose suitable tools with which to implement them.  Such techniques range from high-level work management practices, for example the many variants of agile methodologies, to low-level code development matters such as configuration management strategy.
• Quality plan.  How will you ensure that (every release of) the application meets user needs?  Typically, a combination of strategies is necessary, ranging from code-level practices such as test-driven design and continuous integration through to quality assurance practices based on the principles of Human Interaction Management.

Let's note some things about these documents.

First, such documents are required whether you are implementing business processes via a BPMS, implementing business processes via Model-Driven Architecture, or implementing anything else at all.  Whether you are building a stock trading system for a bank or an embedded device driver for a mobile phone, software construction has certain general principles that must be adhered to.  The rise of agile thinking, and the associated rise of "New Methodologies" designed to handle requirements changes on the fly, have led many people to abandon tried-and-tested software construction practices as old-fashioned.  Beware of throwing the baby out with the bathwater!

Second, the documents do not have to be long.  Rather, the shorter the better.  As long as they cover the required ground, and are consistent with your actual working practice, 3 pages are always preferable to 30.

Third, if you are not going to keep these high-level documents up-to-date, you may as well not bother creating them at all.  Anyone who has ever developed any software at all knows that it always seems to be a moving target.  Everything changes under your feet from the very first day.  In the white heat of a development project, the extra effort required to maintain high-level documents such as those described above may seem like the last thing you should be worrying about.  But actually it is almost the most important thing you can do.

TAKE AWAY

The high-level software project documents described above are  not only your personal bomb-proofing - the evidence, if things go wrong, that you personally acted as a responsible professional - but also the bomb-proofing for the project as a whole.  Armed with current, complete high-level documentation it is possible to salvage many a software project that appears doomed - certainly such documents make it far easier to hand the work over to a new outsourcing supplier, or take it back in house.  Without them, you are working blind.

My first step when engaged to rescue an ailing software project is to make sure that the documents described above are in place and up-to-date - creating them from scratch if necessary, as a retrospective exercise.  Before you can make decisions about where to go next, you need to know where you are now.

In the next postings to this blog I will say more about how to guarantee the success of a software development - and show, in particular, how the often-overlooked Quality Plan can be used to make life easier for everyone.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

To do this, you must at a minimum be able to understand the work that is being carried out.  More accurately, one needs not only to understand the work, but further, to understand how to do the work well.  Then you can step in if it is not being done in the most appropriate manner.

So here is the first step towards outsourcing success: make sure that your own organization has a professional development capability.  Even if this consists only of a couple of individuals, it is essential to have people in-house who are genuinely capable of judging the deliverables provided by an outsourcing company, and positively influencing the way in which the work is being carried out.

Unfortunately, it is getting very hard nowadays to see the wood for the trees in enterprise IT.  A recent Computer Science graduate, for instance, is likely to lack the key knowledge and skills required to make the most of an increasingly complex development landscape.  As I described in the first post in this series, the current profusion of acronyms is enough to bewilder anyone, and it's getting worse daily.

So it is useful to step back from the sea of neologisms, and divide the sorts of skills required into levels that are independent of any particular technology or methodology.  This can be done as follows.  Here is a graded series of skill levels that a software developer should rise through on their way to becoming a modern professional:

1. Language.  This is what someone will typically know coming out of University, or after having taken certifications in a programming language.  It consists of knowing the core library classes/functions of the language(s) concerned, as well as techniques for exception handling, reflection, concurrency, and so on.  Note that superficial knowledge is not enough here - to write a language well, or code review other people's work, you need to know and understand all the features that are available out of the box.
2. Language skills.  Once you know a language, you must then learn how to use it.  This involves stylistic aspects - the way you structure, format and document code. It also involves technical aspects - how to use concurrency safely, pessimistic rather than optimistic programming, how to instrument code to permit analysis/debugging, the different forms of collection object and their uses, proper management of loading/binding, and so on.
3. Design patterns.  Since the late 1980s it has become accepted practice to utilize standard abstraction techniques when writing code, mainly for maintainability but also for code quality and productivity.  The key reference here is the "Gang of Four" book ("Design Patterns: Elements of Reusable Object-Oriented Software" by Gamma, Helm, Johnson and Vlissides), which uses C++ and Smalltalk for its examples.  The GoF book has inspired a decade of further research. A professional programmer must know not only the 23 design patterns from the GoF book but also many additions to and enhancements of these patterns. They will also understand when and how to use each pattern, and how to refactor code - to restructure it to conform more (or less) closely to specific patterns, or just to improve its elegance and readability, without changing its functionality.  As well as for programming, there are also design patterns for enterprise application architecture - the key reference here being Martin Fowler's book, "Patterns of Enterprise Application Architecture".
4. Frameworks.  No-one now writes enterprise applications without using an abundance of frameworks - code libraries that embody best-of-breed solutions to common technical problems. Frameworks are based on design patterns, and you cannot properly understand how to use frameworks unless you first gain confidence with application of design patterns. Many frameworks are open source and can be used to build a product intended for commercial sale.  Frameworks often have associated domain-specific languages such as XMI (XML Metadata Interchange) and OCL (Object Constraint Language).
5. Standards. Any software going into a corporate environment must interact with other software in a standardized way.  Hence such applications need to conform not only to accepted standards but also to emerging standards. For instance, any Java-based application that talks to a content management system must be aware of JSR-170 and JSR-283, which are standards for communicating with such a system via the Java language. There are many important standards pertaining to application development, some like the above arising from a language-specific community, and others (such as Topic Maps for XML documentation of linked content, or DocBook for technical documentation) from non-language-specific bodies such as ISO and W3C.
6. Work Management.  Any professional developer should have some grasp of the various possible approaches to managing a software project, ranging from traditional waterfall techniques through to the many different agile methods and techniques.  There are also design patterns for work management - the key references here are "Organizational Patterns of Agile Software Development" by Coplien and Harrison and my own book "Human Interactions".  I will have a lot more to say about this in future postings, since agile techniques are key to the successful management of outsourced application development - whether or not the project is officially viewed as being "agile".

TAKE AWAY

The first step towards successful outsourced application development is to make sure you know what is going on.  This means that your own staff must be able to understand technical matters at least as well as your outsourcing supplier.  To achieve this, you must ensure that your organization possesses people with skills at all the levels described above.  Some of these people will have only skills to a certain level - but you will need at least some people who have all the skills right up to level 6.   The number of such people required depends on the size of your organization and the number of outsourcing projects you have going on.

This is the first step towards outsourcing success and the most crucial.  In my own consultancy work, I provide clients proposing to engage in an outsourcing project with a learning trail that their key technical staff should follow, customized to the particular type of project they are considering.  If their people are familiar with the material already, all well and good.  If not, acquiring this knowledge not only prepares them for the project in question, but is a valuable contribution to their organization's general capability - a contribution that will stand the organization in good stead in future (as well as, of course, being useful professional development for the individuals in question).

Only once your organization has developed such an in-house capability are you in a position to code review the work that is done on your behalf, as well as to engage with your supplier as to how the work is being carried out.  In the next postings to this series I will discuss how to position yourself so that such engagement is not only painless, but viewed as beneficial by both sides.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (2)

In this article, I will discuss a change of which no-one in IT could fail to be aware - the rise of outsourced application development - and its impact on development practices.

Let's start with some typical problems that arise with outsourcing application development.  Here are some real outsourced projects on which I was brought in to consult:

• A medical system that passed all system tests first time - tests written by the developers - but that turned out to consist of unmaintainable spaghetti code;
• A telecomms data warehousing system that ended up completely dependent for maintenance on a single programmer who had written the core code, who became impossible to work with, and who then decided to retire early;
• An online catalogue with a hopelessly impractical database design in which individual queries often ran into thousands of lines - of SQL code, not results - and consequently each took hours to run;
• A retail replenishment system that had to be completely ditched and re-written from scratch in-house (under huge time pressure) due to the poor quality of delivered code.

These are not particularly exceptional cases.  As with anything in life, you get what you pay for.  Yes, you can save money by using staff from less developed economies - but unless you supervise the work properly, which means managing the code development as closely and carefully as if it was written in house - you will end up with only a short-term gain, and quite possibly not even that.

Why?  Are local developers (wherever "local" happens to be for you) always so much better than developers from other countries?

Of course not.  However, there are 2 key issues to be aware of.

First, programmers are human.  Suppose you are a programmer working under pressure for a company on the other side of the world, a company that does not know you personally, on a system that you will probably never see again once it has been delivered.  Being realistic, you are not going to polish every line of code until it gleams.  You are going to produce as much code as possible, as fast as possible, to the minimum acceptable level of quality imposed by your managers - or to a lower level, if you can get away with it.

Second, outsourcing companies cannot afford to encourage their development staff to invest time in personal development - in deepening and broadening their skills.  Such companies compete intensely on price.  They may be willing to pay more for senior developers, for whose services they can in turn charge more to clients, but they are generally unable to afford to sponsor the increase in knowledge and experience required to turn a junior programmer into such a person.  This means that high-level skills are acquired and applied piecemeal, if at all, by developers working in less developed economies.

As a result, there is an endemic culture of low quality code in outsourcing companies - a culture that means many apparently low-cost systems commissioned from outsourcing companies are in fact unmaintainable, or may not work properly even if they pass all their system tests.

TAKE AWAY

Does your organization rely on outsourced system development for mission-critical processing?  If so, the poor quality of such systems could end up as your undoing.

Coding standards are dropping across the IT industry - even where you would least expect it.  I have seen code written by senior developers with an outsourcing background as part of an extended interview process.  The purpose of writing this code was as an example to demonstrate their abilities, and hence to secure highly desirable work in the West.  Hence one assumes the authors invested as much time and trouble as they could into the example code.  However, while the code they produced did seem to work, the poor quality was obvious at a glance.  The code had been written with such disregard for conventions of style, documentation, etc that maintenance would have been a nightmare.

The authors of this code were all the cream of the crop - highly educated, with experience working on projects for companies such as Google and Amazon, and earning a very respectable amount already - typically around 40 USD/hour, in India!  The problem was that they had never been through the kind of induction process that used to be standard in every professional software development organization.  They did not know even what expectations to set of themselves.

In the next postings to this blog, I will discuss the nature of a professional development induction process, and how to re-introduce it in a world of outsourced application development.  I will also show how to ensure that code delivered from an outsourcing organization is of high quality.

There's nothing wrong with outsourcing application development - but you have to manage it the right way.  If you'd like to find out how, stay tuned to this blog.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (5)

• Analysts have a lucrative business in providing feature comparisons, advising vendors which new functions the market would be most interested in, and helping organizations select from the bewildering array of products on offer;
• Consultants have built a niche helping organizations choose, implement, and maintain these beasts - what a French consulting company with whom I once worked referred to disparagingly as "pousse-bouton" work;
• Enterprise architects charged with business process implementation are able to divest themselves of responsibility by laying it on the shoulders of software product vendors and their promises;
• Software product vendors have found a new market, often for a mixed bag of legacy or bought-in products that they can assemble, revamp, and re-purpose as a BPMS suite.

Nevertheless, it can't last forever.  The major vendors in IT are already owning up to the imminent death of the BPMS implicitly, not only by incorporating BPM functionality into monolithic middleware suites (Oracle, SAP, IBM, etc) but even into the operating system itself (Microsoft).  As I pointed out in my last post, published research from IBM shows how to implement business processes without a BPMS at all.  And here is what David Frankel of SAP has to say about using Model-Driven Architecture (MDA) for business process implementation on bptrends.com this month:

Model-Driven Service Orchestration

Often times, developers trying to get a handle on MDA approach it with the thought that they have
to use it to generate the components of their applications, or at least to generate fat skeletons of
the components, to which programmers add some finishing code. Then they run straight into the
last mile struggles with legacy back end systems that hold so much of the data in non-normalized
form.

Some of the most successful approaches that I’ve seen take a different route to nailing down the
last mile with MDA. They use service-oriented architecture (SOA) to present existing application
functionality as a set of service components whose interfaces are highly normalized via
consistent use of service interface languages such as Web Services Description Language
(WSDL). They use modeling to define specific orchestrations of the services. They walk the last
mile by compiling or executing the orchestration models.

In sum, this approach deals with the last mile problem that non-normalized back end systems
pose, by using SOA to provide a normalized intermediate tier that model compilers and model
execution engines can target in a consistent fashion.

In other words, Frankel is saying that you can use MDA to execute mechanistic business processes via SOA.  Draw your business process models, using whatever notation you like as long as it can generate code - UML is fine, for instance.   Then connect your code to services via WSDL.  That's it.  You can use a BPMS if you wish.  But you don't need to.

TAKE AWAY

Some people are attached to BPMS tools because of the wide range of features they typically offer - for messaging, reporting, configuration, and so on. However, tempting as such goodies may be, their use is not in any way related to increased business success.

In fact, there is a strong argument that adoption of multi-functional workplace tools has exactly the opposite effect.  A wide range of features and options often does little but distract people and muddy the waters.  Being bombarded by information from every side and in every form imaginable is a primary cause of stress - when you add multiple forms of messaging to multiple forms of reporting you get what I call "network overload", which is even worse than information overload.  On top of this many organizations are now using sophisticated workplace tools to transfer system administration responsibilities that properly belong to the IT department - authentication and authorization, for example - to end-users who are neither adequately qualified nor adequately resourced to handle such tasks.

What I see is that the net effect of introducing ever more highly powerful applications into the workplace has simply been to obscure individuals' true business goals and responsibilities.  How many people feel that they are running to stand still these days?  Or even just trying to keep their feet on the ground in the face of a tidal wave of information from all sides - and the demands for response generated by such information?

What the IT industry should be delivering is agile tools - agile in the sense of just-good-enough.  On this basis, the rise of programming techniques based on Model-Driven Architecture will, for many organizations, make a BPMS unnecessary.

For a start, an MDA approach is far closer to the original Third Wave ideal for BPM than the current generation of BPMS tools, since mainstream MDA tools such as Eclipse EMF are based on a simple, general, robust underpinning - unlike any current BPMS products!  It could be argued that the OMG MetaObject Facility that underpins EMF is the closest we currently have to the sort of generic framework envisaged by Smith and Fingar in their seminal 2003 book.

More importantly, MDA makes it possible to define your business goals, then implement processes to support them as quickly and simply as possible - and go round the loop again each time the goals change, as they do continually.  So why do you need anything else?  KISS, as they say.

Happy Easter.

Posted by keithhb in Business Process Management • Management • Office Applications • Security • Service-Orientated Architecture | Permalink | Comments (8)

And it's not just me saying that - it's IBM.

Last week's posting to this blog explained why a first step in building a service layer in your enterprise is to ensure that you grasp the general shape of a modern development infrastructure. In other words, you must ensure that your organization possesses a mature and up-to-date software development capability. This is not as simple as it sounds, not only because there are a huge range of competing technologies out there, but also because vendors of middleware systems are doing their very best to spread FUD (Fear, Uncertainty and Doubt) in the marketplace. Haven't got a BPMS yet? Not doing BAM? No enterprise Business Rules Management System? Tch tch tch.

In fact, many BPM practitioners - and certainly most BPM thought leaders - agree that BPM is a management discipline, not a set of software technologies. And this view goes not only for pure BPM, but for supporting applications like BAM, BRM, CEP, and so on. Technology is simply an enabler of management practice. Get your practice right, and you can often do all this kind of thing very cheaply and easily without need for any big iron middleware at all.

Long-time readers of this blog will remember that, as a simple illustration of this, I often mention the number of monthly downloads of JBoss application server - generally over 200,000 - and point out how this by any reckoning this must dwarf the total number of BPMS installations ever performed, anywhere, at any time. Last year I spoke at the largest BPM conference in Europe and the largest programming conference in Europe. Attendance at the former was in the hundreds. Attendance at the latter was in the thousands.

So why does low-level programming still rule the roost - is it about price? Hardly. Only recently has the success of the open source movement made low-cost or free development tooling a reality for enterprise-level software construction. Further, there are now various free and open source BPMS products available. What it's about is functionality.

Taking the Java world as an example, all the advantages claimed for applications such as a BPMS are now available out of the box for plain old Java programmers. There are low-level technologies that automate process implementation from models - see the IBM white paper mentioned below, for example. Business Activity Monitoring, aka BAM, can be handled very well using aspect-oriented tools such as AspectJ in combination with a dynamic configuration framework such as Spring. Business rules can be implemented, and changed on the fly, using a combination of the Strategy design pattern and flexible deployment technologies such as JMX. These are only examples - there are many more ways to build everything you need for software support of BPM using mainstream programming tools that (if you wish) are available for free and as open source.

BPMS vendors and those with a vested interest in the adoption of such systems (for instance, vendors of associated technologies such as business rules) will no doubt respond to this assertion with the rallying cry: "You may be the kind of techie who just likes programming, but our systems place control in the hands of business people, permitting true business agility".

Oh, come on. As mentioned in the last post, you will always need IT people to manage the configuration and deployment of business processes. No business person is ever going to be able to change anything in a BPMS or associated system - and if they have permission to do so, it ought to be taken away!

What a business person does have every right to expect is for it to made abundantly clear to them, in a plain manner suitable for a non-technical stakeholder, what their systems do. Then there is a proper basis for discussion between them and their IT staff on how the system can be configured to behave differently in future. Then their IT staff can go away and implement the requirements of the business.

Over the last 15 years, I have seen far too many cases in which an enterprise application vendor's promises of tech-free administration, together with a sexy user interface, led to quite disastrous mis-configuration by end-users. Some of these cases would make anyone blanch. And once you have been bitten by the business consequences of such unfortunate meddling by people who have not been trained to understand the consequences of their actions, you will be quite happy to leave the IT department in control of back-end systems forevermore. Business people, give the techies your requirements and let them "make it so"! It's their job, not yours, and everyone will end up happier.

TAKE AWAY

I think it is inevitable that BPM systems and programming technology will converge - and that this convergence will be driven by the programming community, not by the BPM community.

For a start, best-of-breed programming tools now possess a degree of smarts to which no BPM system even comes close. This is partly thanks to the emergence of software design patterns during the early 1990s, and the consequent rise of frameworks based on such patterns, but also due to a number of breakthrough innovations in the last decade such as model-driven development, test-first design and aspect-oriented programming. Modern programming tools are the result of years of massive-scale community effort, and incorporate thinking about software that easily surpasses the best efforts of any application vendor.

Further, since you need IT people to make changes to systems, why not let them do it using the tools they know and prefer?

To conclude, here is an extract from a recent IBM white paper:

Once BPM systems are implemented, they are very hard to change because they are engineered as software development solutions that are linear and rigid or because the monitoring solution derives from process models. Solutions derived from processes are flexible but not comprehensive enough to include the nonprocess metrics needed to represent the full state of a business. Thus, a BPM approach not based on models can fall short of fully meeting business needs.

The abstraction of the BPM solution to higher-level models, as we propose, overcomes the shortcomings of BPM alone. It enables business analysts and system architects to contribute directly to the solution. The MDD approach to BPM means that business goals can be defined independent of an information technology (IT) platform. Business-level models either provide linkage to or can be automatically transformed directly to IT-level models using transformation routines. MDD can quickly reflect changing business goals and monitoring needs through models.

Model Driven Development for Business Performance Management, IBM Systems Journal, Volume 45, Number 3, 2006

Like these IBM researchers, I believe that BPM system implementation in future will be an end-to-end programming effort driven by modelling. My own recommendation if you seek a software infrastructure for BPM is to leverage state-of-the-art modelling tools such as Eclipse EMF, together with a low-level code library for the mechanics of state machine implementation (preferably a free and open source solution such as jBPM). This is not only a practical and low-cost route, but also one that satisfies the need among business people for an object-based description of their processes.

So would you like to discover how to prepare your own enterprise for implementation of such solutions? Then stay tuned to this blog.

Posted by keithhb in Business Process Management • Service-Orientated Architecture | Permalink | Comments (7)

This blog has often addressed leading edge techniques for the integration of business processes with an enterprise IT backbone. A particular focus has been "human-driven" business processes - those in which people not only enter data and record decisions in online forms, but where they also collaborate with each other and leverage business intelligence to create and innovate business solutions.

In recent entries I have articulated a top-down business process management methodology which addresses the most pressing needs of the modern enterprise - to acquire the agility necessary for survival in a globalized, wired marketplace, while simultaneously complying with statutory regulations and company policies, all within a safely controlled management hierarchy. Here is the methodology in outline:

1. First draw up a process architecture, to unite business goals with business processes.  This is a sine qua non - unless you start here, you will be building a house on sand. As discussed often in this blog, goals are the true foundation of all business activities.
2. Next apply Human Interaction Management, to make best use of the humans in your organization, at all levels of the organization chart - not in order to downsize your people away, but rather in order to leverage the skills you have on board.  Only via HIM can you gain the dual advantages of structure (for efficiency) and agility (for responsiveness).
3. Use BPM/workflow to improve your performance of mechanistic work - but be aware that there are no magic bullets to remove real-world complexity!  The idea that BPM would make it possible for business people to change mechanistic processes on the fly is a complete myth.  The IT department are going to stay involved for the duration, and when you want a new version of a mechanistic process you will need to ask IT people to draw it up, IT people to ensure it complies with regulations, IT people to test it, and IT people to deploy it.  Agility is for human-driven processes only - it is the province of HIM, not of BPM/workflow.
4. Finally (and only at this point should SOA enter the picture), look at all the processes you have defined - both human-driven and mechanistic - and ask: which of these could make use of services?  Then build the services you need, not those that the IT department suggests may be quite handy.

This methodology goes by the name of Goal-Oriented Organization Design, aka GOOD.

In previous posts I have discussed most techniques required to implement GOOD, and shown how tooling for BPM and SOA is evolving to meet these requirements. However, the area to which so far I have devoted least attention is the final level, which from an IT specialist's perspective is where the rubber meets the road. How should one go about the provision of a service layer in the enterprise?

There are several related aspects to such work. No longer is it appropriate simply to put together a team of designers, coders, testers, and technical authors, add a project manager and start building systems. The world is full of new deployment frameworks, application frameworks, architectural techniques, programming techniques, communication technologies, ...

Anyone that starts out on an SOA programme without grasping the general shape of a modern development infrastructure runs a serious risk. They are likely to find out part-way down the line that they have not only wasted a large amount of their organization's resources by failing to leverage modern best practices for development, but that what they have spent so much time and effort building is not in any way future-proof and hence only ever going to have a limited life.

Of course, a site such as ebizq.net is full of advice on aspects of SOA implementation. There are also fully-fledged SOA methodologies available from various vendors and even analysts, and no end of consultancy and training services on offer to support it all.

However, many people that I speak to are looking for something much simpler: a simple, clear description of the capabilities required to build a service layer, and roughly how these capabilities relate to one another. The SOA marketplace is chock full of arcane buzzwords, which often describe techniques and technologies that do not so much compete as overlap in ways that confuse almost everybody. This serves to disguise the shape of what actually needs to be done by an organization seeking to build a set of services.

TAKE AWAY

Suppose you plan (as most organizations do) to develop code in Java, either in place of or along with code in .NET. How well do you understand the different offerings of such component technologies as SCA, JBI, J2EE, JMX, OSGi, and JPF? What about new proposals such as JSR-277, or vendor-specific technologies such as the SAP Composite Application Framework? Which of all these are Java-specific and which ones can be used to integrate Java with .NET components?

Most people are struggling so hard with the subtle differences between these acronyms that they have no mental space left for getting the perspective required to make strategic decisions.

In this next series of blog posts I will try to un-muddy the waters, and talk at a high level about the kinds of things you need to think about if your organization is planning to implement SOA on any scale. If you have difficult decisions to make about technology implementation for an enterprise backbone,