To understand the pattern fully, it is necessary to interpret the diagram from three different perspectives:

1. Management
2. Participant
3. Solution

Let's start with the management perspective. This is based on the Human Interaction Management (HIM) principle known as separation of control.  Part of this concept is to divide management responsibilities into executive control (setting goals and responsibilities) and management control (monitoring and supporting the work itself).  Executive control comes from above - management control comes from within.  We also need the HIM concept of REACT, an acronym describing the breakdown of any work activity into 5 stages:

1. Research - find out what you need about the domain;
2. Evaluate - apply this information to the work in hand;
3. Analyze - decide from the options available;
4. Constrain - plan the work;
5. Task - do the work.

There is a lot more to separation of control and REACT than suggested here, but that will do for now.  Applying these ideas to the diagram:

• At the overall level, the Overall Sponsor is the Sponsor Role embodying executive control, while the Domain Sponsor and Overall Design Authority together embody management control. The Overall Design Authority has the Research, Evaluate and Analyze activities, while the Domain Sponsor has the Constrain activity. The Task work is not shown as a single activity, but split between Domain Sponsor and Overall Design Authority, and consists of work assignment and work validation respectively.
• At the domain level, the Domain Sponsor is the Sponsor Role embodying executive control, while the Domain Manager and Domain Design Authority together embody management control. The Domain Design Authority has the Research, Evaluate and Analyze activities, while the Domain Manager has the Constrain activity. As above, the Task work is not shown as a single activity. In this case it is split between all three Roles Domain Manager, Designer and Domain Design Authority, and consists of work assignment, design work itself, and validation of that design work respectively.

As the process unfolds, the Role Activity Diagram will be refined to include sub-components of each high-level activity depicted at the initial stage.  This can be done without contravening the controls established at the start - the pattern is fractal, i.e., it can be applied at a more and more detailed level without change to its structure.

The second way to interpret the diagram is from a participant's perspective. There are three Roles at the overall design level, and another three at the level of each specific design domain or discipline (database design, user interface design, security engineering, and so on). There is a degree of correspondence between these levels that indicates that the model is natural:

Overall design level:

1. Overall Sponsor: Acknowledges that the problem is real and kicks off the solution process
2. Domain Sponsor: At the request of the Overall Sponsor, initiates work in each of the design domains, coordinates the separate streams of work, and reports on progress
3. Overall Design Authority: Responsible for the integrity and quality of the whole design (this Role gives permission for the integration of the specific solution)

Domain design level:

1. Domain Manager: At the request of the Domain Sponsor, assigns expert resource to the problem as it applies to a specific domain, then facilitates and supports this work
2. Designer: Participates in the collaborative activity to develop a domain-specific component of the solution
3. Domain Design Authority: Responsible for the domain-specific aspects of the integrity and quality of the design (this Role gives permission for the domain-specific components of the solution to be released for integration into the overall design)

Different Roles, such as 1 and 3 at each level, may well be performed by the same person. However, since Role Activity Diagrams do not include Users, this is not shown explicitly. What we gain from the Role Activity Diagram with respect to process participants is an overall context for each person's work. This is done in a simple and effective way by the Role Activity Diagram, from which it is immediately obvious how each person's work relates to that of the others, who communicates with whom, and who is dependent on whom.

For instance, it is clear that a domain solution cannot be submitted until each component of that solution has not only been prepared, but has also been "Declared Complete" by the corresponding Domain Design Authority. Further, it is not even possible to submit a domain solution component that has not been approved. Submission can only be done in tandem with the Domain Design Authority, as part of a single interaction that also involves approval - a review meeting, perhaps.

TAKE AWAY

Complex problems - such as large-scale software development - do not have to have complex solutions.  The way forward is to apply abstraction: remove distracting detail, and generalize the situation, so as to arrive at clear and simple patterns underpinning all such situations.

The pattern shown in the Role Activity Diagram is such an abstraction.  In HIM terms, it is a collaborative transaction.

Next time I will explain the third and final way to read the Role Activity Diagram - from a solution perspective - and show how to get started with applying these principles in order to safely manage the development of your own software solutions.  No matter how large and complex your development process appears to be, there is always a simple way to manage it safely.  You just need to look at things from the right angle.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

The trick is to recognize the reality of how people work together - and use that as the basis for a structured approach to managing their activities.

Let's take a typical scenario. A design team is in the middle of a large concurrent design project, and an unexpected issue has arisen - sufficiently serious to warrant specific management attention, and potentially impacting several design areas (database, user interface, security, and so on).

Meeting to discuss the problem, team members realize that a new sub-project is required to manage the issue. Initially they have no plan or structure for this new sub-project. The input to the sub-project is itself a problem definition that may require further investigation and development. However, there are a number of significant things that they can say from the start. They may have none of the detailed elements of a process, but they are able to recognize a pattern. They know how collaborative problem solving generally takes place.

The pattern they recognize for collaborative problem solution has specific aspects that allow us to model it, at a high level, as a process:

1. The people involved will perform certain Roles, each with their own goals and responsibilities. There will be a number of participants in the solution process, with skills in different domains - database design, user interface design, security engineering, and so on. For each type of contributor, there may well be distinct phases of activity: problem definition, concept design, detailed design and testing, integration and testing, and so on. Hence, the sub-project is in fact composed of multiple distinct streams of activity, one for each domain. The contributions of all these have to be coordinated.
2. The eventual deliverables will meet certain preconditions. If successful, the output of the sub-project will be a solution, probably consisting of a set of solution components in different domains. The solution components in each domain will include models and tests that demonstrate that the solution solves the defined problem. There may well be interface specifications and other supporting documentation. Among all these objects there will be varied dependency relationships. Hence, the components of the solution both at domain level and overall are an interdependent set that must be protected from further change unless the whole solution is revisited and retested. The set of solution components can be viewed as a transaction - somewhat like a database transaction, although subject to quite different rules in which all sub-project participants commit jointly to the resolution of the issue.
3. The process participants will interact in a way to ensure that their individual contributions combine to produce overall deliverables of the highest possible quality. The components of each domain solution must be made available to an appropriate person for integration into a unified domain solution. Similarly, the domain solutions must be made available to an appropriate person for integration into a unified overall solution. The individual domain solutions and the overall solution must all be validated and certified by an appropriate authority, and this activity must be linked to the aforementioned integration.

With this analysis in mind, we can use a Role Activity Diagram to represent the sub-project as a whole:

TAKE AWAY

You may be reading this because you are responsible for a large-scale software development. And you may be wondering how the diagram above is going to help you. Is it the magic bullet you need? If so, why?

If you do not find the diagram above self-explanatory, this is because it draws on concepts from Human Interaction Management. Before you can understand the diagram, you need to understand the concepts.

In the next instalment of this series, I will explain 3 different perspectives from which the diagram can be viewed:

1. Management
2. Participant
3. Solution

Armed with this understanding we will start to see how even the thorniest - and scariest - problems in large-scale software development can be tamed. Complex problems can be broken down into a set of simple ones, so that management controls can be applied and the required solutions delivered.

If this sounds like what you need, tune in next time - and if you can't wait, you can always get the book :-)

Posted by keithhb in Management • Security • Service-Orientated Architecture | Permalink | Comments (0)

In the last posting to this blog, I wrote that most change management procedures as documented in a Quality Plan give "absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design."

To deal properly with concurrent design, one must somehow support the unpredictable impact across the entire project of design changes originating from one particular team.  In other words, one must stop being scared of change - trying to prevent it, deny it, or limit it.   Shift happens, and success means accepting that change is a natural part of the software development process.  One must institutionalize continual decision-making on what to do next.

This will break down badly if such decisions are imposed from above, since engineering is a highly complex discipline.  Put simply, decisions that do not involve those working at the coal face will usually be wrong.  So all concerned need to share in the decision-making process.

Agile methods have their root in this understanding.  However, talk to enough software project managers and you will find that agile projects tend to run into trouble when they are over a certain size.  It is hard to define the threshold precisely, but from experience I would put it at somewhere around 30 developers.  Why?

Here is a quote from Martin Fowler, generally accepted as one of the key authorities on Agile methods, taken from his seminal article on Continuous Integration

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

Here is the source of the problem in a nutshell.  Most agile practices assume that the mainline is shared between all developers.  This works fine for small projects, but is totally inappropriate for larger projects, where concurrent design is employed and as a result the work is effectively structured as a set of inter-related sub-projects.  Such sub-projects may diverge widely at times - some may never make it into the mainline at all, while others may only ever contribute part of their own output to the final deliverables.

A good comparison is with school class sizes.  Teachers tend to struggle when class size grows beyond 30.  Well, so do software project managers.  In both cases, the solution is to divide people into groups that are semi-independent, and can work at their own speed, making their own choice of subject matter and how far to pursue it.

In my view, there are 2 reasons why the software development community has not properly taken this on board:

1. Failure to understand configuration management techniques.  This is not really excusable, given that there are simple guidelines that anyone can follow to make use of multiple concurrent branches entirely safe.  Laura Wingerd of Perforce Software gives a great explanation of how to manage what she calls the Flow of Change: How Software Evolves (book excerpt) and presentation on same material.  The techniques she explains can be used with any modern software code repository, such as CVS or Subversion.
2. Failure to manage continual decision-making on what to do next.  This is more excusable, since patterns for such collaborative work have only recently emerged, in particular with the discipline of Human Interaction Management.

TAKE AWAY

Are you responsible for (or just working on) a large software project?  If so, you need to think seriously about the consequences of concurrent design.  Here is what IBM has to say (Best practices for software development projects, 10 August 2006):

Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion.

It makes no difference what you are building, or how - whether you are automating business processes using Eclipse MDA, automating business processes via a BPM Suite, or developing a stock trading system.  The IBM article above goes on to list as the first practice crucial to success:

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Exactly.  However, you won't deal properly with the continual changes arising from concurrent design by trying to impose a rigid set of procedures.  What you need are organizational patterns naturally suited to a dynamic, evolving work environment.  In the next instalments of this series I will say more about such patterns, and how to implement them.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (1)

Here is a typical description of a Quality Plan, selected at random from the Web:

Quality Plan Description

The Quality Plan describes how a developer's overall quality process guidelines will be applied to a project. It defines what is meant by the various quality-related tasks in the Project Plan.

For example, a developer's quality manual may describe a review process for ensuring that delivered software meets requirements. The Quality Plan for the project tailors this general definition to the project at hand, specifying items such as who generates the requirements, what form these will take, who reviews them, etc. Thus, when a task in the Project Plan reads "Review and update requirements," the Quality Plan defines what will be done, and who will be responsible.

Quality Plan Contents

The Quality Plan outlines how you will build quality into the software and documentation. The dates assigned to key tasks in the Quality Plan are entered into the project plan.

The Quality Plan describes:

• How you control changes.
• How you ensure that the product meets the requirements (validation).
• How you ensure that the product works properly (verification).
• How you track multiple development builds of the software to avoid confusion (configuration management).
• How you plan for and execute testing, both incrementally during development and for the entire product before delivery.
• How you track and resolve defects.
• How and when you conduct design reviews, code reviews, walk throughs, reviews of test scripts, reviews of test results (for example, is 100% of all code checked, or only the most complex parts?).
• Definitions, methods, and criteria you use to determine whether the software has passed each review.

At first glance, this seems OK.  However, it is in fact rather a limited approach to quality planning.  The problem with such a definition is that one could get away with almost anything (or nothing) in the Quality Plan document, since the only expectation is to have some form of test and review during the life of the project.  Quality in its classical definition means "fitness for purpose".  So the real question that should be asked of a Quality Plan is this: how does it ensure that the software delivered is fit for its purpose?

A more thoughtful approach to quality planning is that of the PRINCE 2 method developed by UK government, and now an internationally recognized approach.  Here is a typical PRINCE 2 approach to quality planning:

Purpose

The purpose of the Project Quality Plan is to define how the supplier intends to deliver products that meet the customer's quality expectations and the supplier's quality standards:

• Does the plan clearly define ways in which the customer's quality expectations will be met?
• Are the defined ways sufficient to achieve the required quality?
• Are responsibilities for quality defined up to a level that is independent of the project and project manager?
• Does the plan conform to the corporate quality policy?

Composition

The Project Quality Plan should contain:

• Customer quality expectations
• Acceptance criteria, a prioritised list of criteria for the final product(s) that must be met before the customer will accept the final product(s).
• Quality responsibilities, who is responsible for each of the aspect of quality of the final product(s)
• Reference to any standards that need to be met
• The quality-control and audit processes to be applied to project management
• Quality-control and audit process requirements for specialist work
• Change management procedures
• Configuration management plan
• Any tools to be used to ensure quality.

Derivation

• Customers quality expectations and requirements
• Organisational or programme quality management system and standards
• Configuration management and change control requirements

Such a description will produce a more useful (if longer) document.  However, there is still a fundamental problem with it.  The problem lies in the innocuous phrase "Change management procedures".  Typically such procedures comprise some combination of:

• Techniques for issue tracking
• Guidelines for approval of proposed changes.

This gives absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design.

Demands for increased productivity and shorter times-to-market generally force software engineering organizations to adopt concurrent design.  Different parts of a complex application are developed in parallel and then integrated, shortening project schedules. Typical interactions in this type of business process are activities such as agreeing on sub-contract terms, signing off a specification, concluding a project stage, and so on. Each of these may involve several parties, contain processing spread over varied systems on different platforms owned by different organizations, and require a number of steps to complete successfully. Further, although efficiency gains can be achieved via this approach, it carries risks of rework, which arises when concurrently performed work turns out to be incompatible, and an impact of one interaction is that others need to be repeated.

In other words, concurrent design is what Human Interaction Management defines as a typical human-driven process - fraught with instability, since software engineering managers are dealing with activities that are only partly predictable when the process commences. It is a fundamental characteristic of the whole environment that it is organic and dynamic, so hard coded process descriptions are unsuitable. Even if they are supported by tools that make modification of that process possible, the participants are innovative, creative people who need the flexibility to put their skills and experience to use in what seems like the most appropriate way at the time.

There is a parallel situation in the structure of the product itself. A complex software application may appear to decompose neatly into components (and services) but in fact it is a densely tangled set of solutions to a huge and varied set of requirements and constraints. Later modifications only increase the complexity. New connections and dependencies are introduced by tracing the impact of modifications, defining and implementing consequent changes, testing the whole package and restoring the integrity of the product.

TAKE AWAY

Software product managers attempting to deal with applications built using concurrent design generally find themselves tearing their hair out by the time the project is halfway through.  If they are lucky they will be running to stand still.  If not, they will find the project slipping further and further every week.  Failure to properly handle concurrent design is probably the single biggest cause of software project failure.

The solution to managing concurrent design lies in giving the ever-changing nature of a human-driven process the respect it deserves.  In such work, a major part of the workers' duties is to decide on next steps.  This must be done both continually and collaboratively, since there are always new issues to resolve and few of them can be dealt with in isolation.

In the next instalment of this series, I will look at techniques for structuring this "continual decision on next steps".  If you want to make your change management as elegant and efficient as your software code, stay tuned.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (0)