Dealing with change during software development - IT Directions

« Quality is a moving target | Main | Managing large-scale software development »

May 14, 2007
Dealing with change during software development This is the eighth in a blog series on the Future of Programming - and it's time to draw a few threads together. Back in episode 5, How to become an IT professional, I promised to show how to position yourself so that engagement with your outsourcing supplier on how work is carried out "is not only painless, but viewed as beneficial by both sides". It has taken a few more instalments to get there, since first we needed a proper discussion of Quality Planning.

In the last posting to this blog, I wrote that most change management procedures as documented in a Quality Plan give "absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project. This is because most enterprise software projects rely on concurrent design."

To deal properly with concurrent design, one must somehow support the unpredictable impact across the entire project of design changes originating from one particular team. In other words, one must stop being scared of change - trying to prevent it, deny it, or limit it. Shift happens, and success means accepting that change is a natural part of the software development process. One must institutionalize continual decision-making on what to do next.

This will break down badly if such decisions are imposed from above, since engineering is a highly complex discipline. Put simply, decisions that do not involve those working at the coal face will usually be wrong. So all concerned need to share in the decision-making process.

Agile methods have their root in this understanding. However, talk to enough software project managers and you will find that agile projects tend to run into trouble when they are over a certain size. It is hard to define the threshold precisely, but from experience I would put it at somewhere around 30 developers. Why?

Here is a quote from Martin Fowler, generally accepted as one of the key authorities on Agile methods, taken from his seminal article on Continuous Integration

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

Here is the source of the problem in a nutshell. Most agile practices assume that the mainline is shared between all developers. This works fine for small projects, but is totally inappropriate for larger projects, where concurrent design is employed and as a result the work is effectively structured as a set of inter-related sub-projects. Such sub-projects may diverge widely at times - some may never make it into the mainline at all, while others may only ever contribute part of their own output to the final deliverables.

A good comparison is with school class sizes. Teachers tend to struggle when class size grows beyond 30. Well, so do software project managers. In both cases, the solution is to divide people into groups that are semi-independent, and can work at their own speed, making their own choice of subject matter and how far to pursue it.

In my view, there are 2 reasons why the software development community has not properly taken this on board:

Failure to understand configuration management techniques. This is not really excusable, given that there are simple guidelines that anyone can follow to make use of multiple concurrent branches entirely safe. Laura Wingerd of Perforce Software gives a great explanation of how to manage what she calls the Flow of Change: How Software Evolves (book excerpt) and presentation on same material. The techniques she explains can be used with any modern software code repository, such as CVS or Subversion.
Failure to manage continual decision-making on what to do next. This is more excusable, since patterns for such collaborative work have only recently emerged, in particular with the discipline of Human Interaction Management.

TAKE AWAY

Are you responsible for (or just working on) a large software project? If so, you need to think seriously about the consequences of concurrent design. Here is what IBM has to say (Best practices for software development projects, 10 August 2006):

Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion.

It makes no difference what you are building, or how - whether you are automating business processes using Eclipse MDA, automating business processes via a BPM Suite, or developing a stock trading system. The IBM article above goes on to list as the first practice crucial to success:

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Exactly. However, you won't deal properly with the continual changes arising from concurrent design by trying to impose a rigid set of procedures. What you need are organizational patterns naturally suited to a dynamic, evolving work environment. In the next instalments of this series I will say more about such patterns, and how to implement them.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture |Digg This|Add to del.icio.us

Comments

Nice post Keith. Blogged about it here.
JT

Posted by: James Taylor at May 15, 2007 02:25 AM