Recently a lot of attention has been paid to the book Wikinomics, by Tapscott and Williams:

Today, encyclopedias, jetliners, operating systems, mutual funds, and many other items are being created by teams numbering in the thousands or even millions. While some leaders fear the heaving growth of these massive online communities, Wikinomics proves this fear is folly. Smart firms can harness collective capability and genius to spur innovation, growth, and success.

A brilliant primer on one of the most profound changes of our time, Wikinomics challenges our most deeply-rooted assumptions about business and will prove indispensable to anyone who wants to understand the key forces driving competitiveness in the twenty-first century.

Based on a $9 million research project led by bestselling author Don Tapscott, Wikinomics shows how the masses of people can participate in the economy like never before. They are creating TV news stories, sequencing the human genome, remixing their favorite music, designing software, finding a cure for disease, editing school texts, inventing new cosmetics, and even building motorcycles.

This is hype, of course. Less breathless appraisals of the book than its blurb above point out some of the book's shortcomings:

A review of this book in the Harvard Business Review states "like its title, the book's prose can fall into breathless hype." A review of this book in Choice recommends the book for "general readers and practitioners," but cautions that the authors "present an optimistic overview of successful collaborations and business ventures", "use unique terms (e.g., marketocracy, prosumption, knowledge commons)", should have given "more consideration [to] the darker sides of human motivation as well as groupthink and mass mediocrity", and "primarily draw on their own observations of businesses and trends for the ideas presented."

Nevertheless, it seems clear that blogs, wikis, et al are helping to bring about a fundamental change in the workplace. They accelerate the process started by the Web itself, namely democratizing the publication of information. One could view such technologies as enablers for the Community of Practice idea that has been hovering on the fringes of business life for the last 2 decades.

Further, the widespread use of such powerful communication tools is helping people realize that communication is not the same as collaboration. In fact, many of the people spearheading this recognition are those keenest to promote the existing communication tools. They seek new collaboration tools to go with their wikis and blogs, since they know that for their beloved wikis and blogs to succeed long-term, there must be a way for organizations to manage the explosion of communication that is resulting from large-scale, one-to-many information publication.

Without some form of control applied appropriately at each level of the organizational hierarchy, the "wiki workplace" will simply descend into anarchy.

TAKE AWAY

The authors of Wikinomics are sponsoring the creation by the general public of an "unwritten chapter", via a wiki that will be transformed into a document for publication at key points. This wiki has some interesting ideas. In particular, the vision espoused therein for Enterprise 2.0 offers a compelling vision for the next generation workplace, suggesting that a "People-Oriented Architecture" (POA) is needed to go with today's SOA.

Regular readers of this blog will recognize a similarity between the "Enterprise People Bus" described therein and the knowledge bus concept outlined in some of my previous posts.

There is no doubt that wikis, blogs and the whole panoply of social software tools are changing the workplace. Whether or not this change is for good depends on how fast organizations take up the collaboration tools needed to bring order to the chaos.

Posted by keithhb in Management | Permalink | Comments (0)

If you have been following my recent blog posts, you may be interested to know that a consultancy client recently evaluated the effects of applying the techniques described in the post on hyper-productivity to fault fixing.

Their conclusion was that when using my techniques for fault fixing, faults were fixed in approximately half the normal length of time, or in other words, that developers were twice as productive as usual.

TAKE AWAY

Most literature on fault fixing estimates that the average developer spends over 50% of their time fault fixing. So if you are looking to improve the efficiency of your development staff, it must be worth considering techniques that double the productivity of this time.

Posted by keithhb in Management | Permalink | Comments (0)

Following my last post, on Hyper-productivity, I thought I should add some remarks about staff levels.

Budget and resource constraints often mean that, even if too many faults are appearing in a software project, it is not always possible just to add staff. However, even if it is possible, it is often not a good idea.

Adding staff works on the seemingly common sense principle that if 10 developers can fix n faults per week, 20 developers can fix something not too far from 2n faults per week. Software project managers often seem to believe that having more people are working on the job will inevitably lead to an improved delivery rate.

Unfortunately, however, this is not true. In fact, adding staff may well lead to a reduced delivery rate.

Many readers will be familiar with Fred Brooks' seminal book on software project management, The Mythical Man-Month: Essays on Software Engineering. The concept of "The Mythical Man-Month" refers to his principle that "Adding manpower to a late software project makes it later". Brooks says this is because adding people to a project has 2 primary knock-on impacts.

First, the newcomers have to be trained in the skills they need, and familiarized with the project itself. This takes up not only their time but the time of other people - often the people who know most, and who are hence most productive. Induction also takes up other resources of various kinds that can then bottleneck the overall work being done by the project.

Second, the number of communication channels increases dramatically with the number of staff. Brooks' formula is that for n developers, there are n(n - 1) / 2 communication channels within the group. For example, 50 developers implies 50(50 - 1) / 2 = 1225 channels of communication.

This second point is a simplification, of course. Many people would argue that careful management can reduce the number of communication channels. However, from my experience, Brooks' formula is useful whether or not it is exactly correct, in that it indicates the scale of the problem. If you observe any large project from the floor you will see that there is a great deal of human interaction taking place, of all different kinds - not just formal meetings, but emails raising issues, questions asked over the cubicle wall, casual discussions by the coffee machine, and so on. It is very hard indeed to quantify this interaction and even harder to assess its value.

TAKE AWAY

It is possible to add staff to a late-running software development project and gain value. However, it must be done very carefully. To be specific, first you must design the collaborative work processes in which the new staff will engage, using the techniques of Human Interaction Management.

Only by this means can you find out what bang you will get for your buck - what improvement in delivery rate you can expect from purchasing extra staff. In fact, only by this means can you be sure you will actually get an improvement, rather than the deterioriation expected by Brooks.

Further, a moment's thought will show that designing collaborative work processes for new staff is no use unless you have already designed the collaborative work processes for current staff - and ensured that current staff are actually engaging in these processes. Otherwise the structured interactions of the new workers will dissipate into chaos as soon as they start working with people already on the project.

It is possible to improve a late-running project by adding staff, but you can't do it by building a house on sand.

Posted by keithhb in Management | Permalink | Comments (0)

In the last few months I have been approached several times for advice about large software projects in which the fault rate is out of control. This is not as simple as saying that faults are appearing much faster than they can be fixed - at certain stages of a project, one expects this. The real problem is more complex, and relates to the nature of large software projects.

It is important to understand in what ways a large software project is unlike a small one. As I often suggest, one can gain understanding here by comparing software engineering with traditional heavy engineering. For instance, engineers who deal with extremely large and complex systems in fields such as aerospace and the military have always been used to projects that go live with a high number of faults, many of which will remain open throughout the entire lifetime of a system. Fault control in such systems is as much about managing faults as about fixing them.

However, this is not to say that one can take a relaxed attitude towards fault control. There are well-established metrics available for the "defect density" one can expect at each stage of large software projects of certain types in certain fields. These metrics allow software project managers to use the current number of faults, the current fault rate, and the current fix rate to predict where their project will be in the future - and understand whether or not these predictions are worrying.

The situations about which I was asked were worrying. Predictions showed that build deadlines would not be met, and that by a certain point in the future, the number of faults would escalate beyond the managers' ability to control. What should one do in such circumstances?

Taking another leaf out of the military's book, there are 3 approaches that should be used in combination: operational, tactical and strategic. If you prefer, think of these approaches as short-, medium- and long-term. Taken together, they provide a means for even large software projects to become "hyper-productive" - a phenomenon well-recognized in certain small teams, but rarely achieved on the biggest projects.

Operational

One needs to do something in the short-term, not only to stop the situation getting further out of hand, but also to raise morale. Developers do not like to work on failing projects, and once your developers start leaving (or just giving up), you are really in trouble. Even if management somehow conceal the true situation from developers - which is never easy - there is the morale of managers to consider as well.

So in the short-term, it is advisable to set up an emergency room - a lab environment in which some of the best developers can work free from disturbance to knock some of the worst faults on the head as soon as possible. It is often surprising how much progress can be made by reducing the amount of interruption people are subject to. People should not be allowed to enter the emergency room without permission, and if the lab fault fixers need to talk to someone from outside, that person should be brought into the room specially.

A manager should be appointed for the lab, a person who is able to create a team atmosphere as well as maintain a sense of urgency. A key part of the manager's role is to ensure that work is allocated in the most efficient manner possible from hour to hour - sometimes from minute to minute. Everyone should be fully loaded, the whole time, and distractions such as checking email discouraged. The aim is to maintain a sense of flow.

Lab work of this kind is exhausting, so the participants should be rewarded with overtime payments and pampered with free, luxury food and drink. However, the work is also rewarding due to the awareness of visible progress and the team spirit that should be engendered. Further, such a lab is a powerful means of knowledge sharing - after working in the lab, a developer will not only have more system knowledge but may well have acquired new ideas and techniques that will be useful going forward.

Nevertheless, this approach is not intended to be a normal part of the project. If it is necessary to carry it on for more than a month, the participants should change regularly - weekly, for example - to avoid burn-out. Lessons can be learnt by both developers and managers from such a lab, but the emphasis should be on carrying these lessons into regular daily practice rather than on sustaining the lab itself.

Tactical

In the medium-term, there are various means to reduce pressure on a project, with which most project managers are familiar. Deadlines and the content of deliverables can be renegotiated with the client. Workarounds can be developed for areas of fault. End-user expectations can be managed, such as when to expect the introduction of particular aspects of system functionality. Training can be planned so that end-users acquire system knowledge gradually, in order to allow time for the corresponding functionality to be released. Easy-to-use techniques can be developed for end-user fault reporting and control.

When proposing such measures to the client(s), one should take care to present a positive picture. Don't emphasize the trouble the project is in! Rather, explain how your advanced project management techniques have identified potential problem areas well in advance, and prepared a range of compensating solutions. The client(s) should come away from such a meeting feeling glad that you are in charge, not someone who might have let the problems go unnoticed until it was too late.

Strategic

However, it is only fair to admit that, if the project has got to this state, someone has slipped up - even if it was your predecessors. Clearly the scope of the work was misjudged, inadequate development techniques have been used, project management has not been optimal, and so on. So it is necessary to take remedial action to get the project properly back on track in the long-term.

There is a wealth of advice available here, particularly with regard to "Agile" techniques. I recommend Jim Coplien's book Organizational Patterns of Agile Software Development. If you can't face a long read, here are some summaries of "the ten patterns that research has found most strongly correlate to business success": PDF HTML.

One should take the time to understand what Agile project management is all about. Unfortunately, though, Agile techniques work much better with small projects than with large ones. In my experience, agile techniques typically have most success in projects with less than 50 staff, and projects with over 100 staff need something else. So here are some techniques that I have found useful in getting large projects back on track.

1. Food
   Provide ample, high-quality food free to all project staff daily. A small investment in pampering of (say) £10 per person per day pays huge dividends in morale and commitment. Don't be mean! The aim is to make people feel valued.
2. Team ownership of code
   You can't act responsibly if you're not responsible for anything. Coders must own their bugs and work as a team to reduce them. Develop continuous integration practices and put up highly visible flags in each team area every morning: green, amber, red to indicate code quality for that team based on automated testing of last night's build. Institute a daily trawl through all open faults by each Work Package Manager with a view to escalating or delegating faults or actions as appropriate. Introduce a weekly quota of fault fixes based on size of team.
   Careful synchronization with the emergency room is clearly necessary here while it is running.
3. Knowledge management
   Build system knowledge. Black box: give developers a user's perspective - terminology, operation, concerns, etc. White box: make sure they understand what all the different parts of the system do. Do both these things as early and as widely as possible to get maximum benefit. It doesn't take as long as you might think - lunchtime seminars could be the way to go, for instance.
4. Testing
   Give developers a means to test the system as a whole (not just to run unit tests). In particular, they need a framework with which they can construct and run end-to-end tests themselves, for example to reproduce informal user fault reports or check particular behaviour relevant to a fault. Helping develop such a framework may be one very useful output from the emergency room.
5. Pair programming
   Pairs of roughly equal ability but mixed knowledge work best. Change the pairs often. Check to ensure that both people are contributing rather than having one driver and one passenger. Convince people to give it a go - nearly everyone finds they like it. In general it is more than twice as productive - i.e., worth doing.
6. Systematic fault fixing
   Most developers in the industry as a whole spend over 50% of their time fault fixing. Yet they never learn how to do it properly - probably because there is no standard methodology or even practices. I have compiled a systematic fault diagnosis procedure based on the research papers and books that are available, and found that for faults that take more than 2 hours to diagnose, it makes a dramatic difference. It has enabled faults to be solved in hours on which experienced developers previously spent weeks and got nowhere.
7. Reduce distraction
   The BBC reported on 13 August 2007 that "Workers are 'stressed out' by e-mails". Regular readers of this blog will be aware of my view that this is a pernicious problem across the board in the modern workplace, and that the solution lies in new Human Interaction Management tools such as HumanEdj.

TAKE AWAY

It is often tempting to push problems under the carpet when they seem to huge too solve. Large software projects that have gone out of control can seem like a many-headed hydra - as soon as you solve one problem, ten more problems appear. However, there is always a way through the labyrinth.

The key is to be systematic - in particular, divide remedial measures into operational, tactical and strategic.

Further, take care to explain what you are doing to others and get them on board - this is key. Developers as well as managers should understand what is going on and what you are doing about it. A typical but very unhelpful approach to managing crisis situations is to take everything behind closed doors. Resist this temptation! People are surprisingly good-natured, even in times of stress, if they are made to feel a valued part of something - as opposed to being excluded from important discussions.

There is a route to hyper-productivity, and it is not hard. You just have to take it in stages.

Posted by keithhb in Management | Permalink | Comments (0)

A theme of my postings this Autumn has been how to work more efficiently - in particular, how to collaborate more efficiently. For example, I described last month how the business world is waking up to the problems caused by email (see Inbox Zero). However, it is not really fair to blame email per se - for two reasons.

First, the problem is wider than messaging. People may finally be realising the business problems caused by email-related issues such as:

• Groaning inboxes
• Failure to group messages naturally by content
• Discussions that fizzle out
• Discussions that fragment among different colleagues
• Discussions that lose their purpose entirely
• ...

However, even these issues are only part of a general problem: the inability to structure high value human work. Other symptoms that are only indirectly related to email (or messaging generally) include:

• Documents scattered all over your file system
• No way of returning to a previous version of a document
• No way of knowing what version of a document your colleagues are using
• Actions that cannot be tracked
• Actions for which you are not sure if anyone has even taken responsibility
• Doing work without knowing what value anyone is getting from it
• ...

The general problem is that people doing high value work may have a great deal of skill and experience; they may be able to get on well with their colleagues; they may be motivated and committed. But this is not enough. Without an understanding of how to work, all these inputs are often put to less than optimal use. This is a polite way of describing things - many people routinely feel as if they are banging their head against a brick wall when trying to get things done in an organizational context.

This brings us to the second reason why it is not fair to blame email per se: the problem is not fundamentally one of technology. The root of the problem is that people go about their business, and their employer's business, without having any useful way of structuring their activities. The means currently on offer are either not supportive enough (for example, time management techniques and calendar software) or too restrictive (workflow and BPM, even in its current evolution towards integration with office systems). None of these offer a robust framework in which the individual can understand who they are working with, on what, when things must be done, and why it is all happening in the first place. Since all these things change every day, it is all the more vital to provide the individual with a way of seeing the wood for the trees.

This way of seeing is not a set of diary appointments or task sequences - it is a network of goals, responsibilities and commitments. For more information, see the Human Interaction Management Web site.

TAKE AWAY

The desktop of the 21st century will not contain menu items and icons pointing to programs that you have installed - rather, it will contain active collaborations that you are engaged in. You will not explore files in folders - you will explore documents that represent your involvement in a process. You will not keep one list of contacts in Outlook and another in Skype - you will just have a list of colleagues, identified by name rather than messaging address.

The workflow and BPM community is unlikely to make this happen, since it represents a seismic shift in approach:

• Away from monolithic server-side applications and towards lightweight peer-to-peer computing
• Away from task sequences and towards human interactions
• Away from the IT department and towards the business person.

The desktops of the future are as likely to come from the games community as the enterprise integration world!

In the meantime, the reference implementation for the desktop of the 21st century is the free HumanEdj software. If you would like a sneak peek at the future, the new version is just out in beta.

Posted by keithhb in Management | Permalink | Comments (0)

I promised in a previous post to explain more about the free HumanEdj software that will be available in a month or two. Why should you be interested?

You might not be - if you are one of the few people who use the Web but not email. But if you are like the rest of us, and depend on email for collaboration with colleagues, then you need HumanEdj. Even if you don't yet realize it.

There is an old joke - for some people, email gets in the way of their work. For others, it is their work. As far back as 2003, the BBC claimed that the average length of time per day spent just reading (not responding to) email was 2 hours. In 2005, the average number of emails received per day was 133 (Radicati Group). Going on observations made at the sites of my consultancy clients, I would be very surprised to find that these averages have decreased since then. Many organizations seem to run on email, even more than the telephone - take email away and they would grind to a halt.

This ancient set of protocols (SMTP, POP, IMAP and X.400), intended originally only for communication among researchers working in the same lab, now underpins the modern workplace in the same way as the electronic funds transfer standards ISO 8583 and ISO 7810 now underpin commerce. When you consider that the above figures are averages, it is clear that many people spend most of their working life doing email.

This makes it all the more surprising that email is so fundamentally broken as a workplace tool. To take just one example, everyone knows how hard it is to find old emails. Filtering into folders is incredibly hard work to set up and maintain, and doesn't usually apply to sent emails anyway. Simple searching of unfiltered emails is not much better, since both techniques are based on searching for keywords in some part of the email - sender name, subject contents, date, etc. Yet keywords are no true guide to the context of the message. Two messages may have the same sender and date, and even contain some of the same words in the subject line, yet concern quite different matters - 2 unrelated shipments from the same supplier, for example.

At the most primitive level, people need a better way of grouping emails - into work processes - and to have this grouping done automatically. People would like attachments to be associated with these processes, rather than scattered all over their filesystem. Different versions of these attachments should be related to one another, and proper version control provided. People also need to see higher-level information such as which messages are still awaiting reply, and which documents are awaiting perusal, both by them and by their colleagues.

Going further than just messaging, a key use of email is to agree on actions with colleagues - so this should also be made clear. Proposals for next steps (effectively, work process descriptions) should be clearly identified, along with information such as which proposals have been accepted and by whom. It should be possible to send progress reports and associate them with such process descriptions. Going further, why not provide an interface to the activities themselves from the messaging system, if that is where the activities were defined?

Related to action management is a common problem in the enterprise workplace - what I call the "global CC nightmare". Someone has an issue that they cannot personally resolve, but they are not sure who is the right person to pass it on to. So they CC a mixed bunch of people, hoping that someone will pick up the baton, and in doing so safeguard themselves against a later accusation of inaction. In effect they clear their own desk, at the expense of creating an unholy mess in which responsibility for the issue is totally lost. It is quite likely that either nothing will get done, or competing efforts to deal with the issue will derail each other. To alleviate this, it is necessary to add "opt-in" and "opt-out" functionality to messaging, so that people can declare their (un)willingness to take part in something to the others potentially involved.

Finally, there is the whole quagmire of Business Intelligence. What sorts of information can be gathered from messaging? It should be clear even from the above brief discussion that a huge amount of highly valuable information about staff activities is currently hidden from a manager's view.

TAKE AWAY

If you want efficiency in the 21st century workplace, email is the place to look. Although many office workers spend a large part of their working day using email, it is neither an efficient tool nor a useful source of management information.

Fortunately, new software tools are emerging. Underpinned by the principles and patterns of Human Interaction Management, the new version of HumanEdj brings to fruition ideas developed in response to a global need for better collaboration. HumanEdj sits on top of your current email system(s), integrating seamlessly with them to provide the higher-level features that make it possible to work better. Further, it is a simple client program, requiring in most cases zero configuration - download and install it, then it will automatically work out your name, connect to your email client, load your address book, and so on.

Despite this, previous versions of HumanEdj have presented a rather forbidding interface to the novice, requiring people to learn and use new terminology. However, the new HumanEdj 3 provides a radically revised and simplified user interface, based on feedback from early adopters - you now get the benefits without having to acquire the theory! If you are interested in trying out a pre-release version, see the Web forum for details and feel free to contact me - by email, of course ;-)

HumanEdj 3 is now available for download. A short demo video is also available.

Posted by keithhb in Management | Permalink | Comments (0)

Business Process Management is generally acknowledged - at least by practitioners, if not by software vendors - to be a form of management, not a form of technology. With this in mind, 2 books I read recently are illuminating on what BPM currently stands to achieve.

Peter Drucker used to say that management contained elements of both art and science - he thought of it as a "liberal art". Despite his pivotal status in the development management thinking, this put him rather at odds with the general current of 20th century work in the field, most of which focused on extending Frederick Winslow Taylor's efforts to develop a "Scientific" approach to management. The ideas of Drucker, Senge, Handy and other such systems-oriented thinkers may inspire people, but on balance they have probably had far less impact on the day-to-day operations of business (and certainly less prominence in MBA courses) than figures such as Shewhart and Deming, who developed Taylor's principles into a set of simple practices via which business people could organize their activities. Total Quality Management, Six Sigma and BPM all stem in the end from Deming's Plan-Do-Check-Act (PDCA) cycle.

PDCA bears only a peripheral relation to true science. It may employ a variant of the Scientific Method in its iterative approach to developing a useful process, but it can hardly be thought of as seeking to uncover any fundamental truths about the universe! However, let's stick with the analogy with science for a moment.

The first of the 2 books mentioned above is "Progress and the Invisible Hand: The Philosophy and Economics of Human Advance", by Richard Bronk. This work typifies a general trend in 20th century ideas about science. From the First World War onwards, disillusion with the potential of science grew and grew, reaching its apotheosis in Rachel Carson's 1962 polemic against DDT, "Silent Spring", which many people consider to have marked the start of the Green movement in politics. Bronk's book epitomizes this attitude, and is particularly wide-ranging. It's central thesis is that Enlightment thinkers were misguided - scientific control over nature, as expressed via the operation of a free market, has not in fact made us happier, and is unlikely ever to do so.

The second book, by contrast, is an impassioned defence of science. "Science and the Retreat from Reason", by John Gillott and Manjit Kumar, argues that any problems caused by science are simply due to misuse and misinterpretation. Rather than being our undoing, science has the potential to solve most evils, if only it were properly funded and its practitioners properly motivated.

The books are not exactly opposed. Both of them can be taken to make the case for greater and more thoughtful government intervention to support the efforts of scientists. However, there is a piece of the puzzle missing, which is this. Science cannot be used effectively without society being ready and able to do so. In other words, there must be an effective mechanism via which new technology resulting from scientific discovery can be put to use for the general good.

TAKE AWAY

What has all this discussion of science and society to do with BPM?

Let's suppose that management is in fact a quasi-scientific discipline. Then BPM in its current form - and the software used to implement it - can be likened to a brutalizing technology such as DDT. A workplace dominated by BPM is one in which innovation is stifled, collaboration is reduced to depersonalized approval routing, and an individual person's duties consist of carrying out particular steps in a pre-defined and mostly automated sequence. This Kafka-esque vision is all that we can possibly build with methods such as Six Sigma and Lean, and tools such as BPMN and BPEL4People. Just as the side-effects of DDT were to lay waste to ecological diversity for decades, unrestricted use of BPM in its current form has the potential to lay waste to healthy workplaces worldwide.

In fact, this is only scratching the surface of the impact in store for organizations that commit whole-heartedly to current BPM methods and technologies. In future postings to this blog, I will illustrate some disastrous end results that may lie in store for adopter's of today's technology buzzword, SOA, and show how SOA has a good chance of completely destroying your IT infrastructure.

This is not the fault of BPM and SOA itself, which are simply technologies - ones that, used appropriately, have every chance of delivering all their promised ROI. However, like science in general, they cannot be positive forces unless applied within an organizational "society" that permits innovation to thrive, collaboration to take place naturally, and individuals to work in a healthy way. What we need is not more systems thinking, but a simple, practical, step-by-step way for people to work together better, both within and across organizational boundaries.

Does any of this strike a chord with you? If so, stay tuned to this blog over the Autumn to find out why SOA is so potentially dangerous, and how the techniques of Human Interaction Management can be used to avoid meltdown.

Posted by keithhb in Management | Permalink | Comments (2)

First, I apologize to regular readers of this blog who have been wondering what has happened to it recently. It has been a busy Summer! In particular, development of the next version of HumanEdj has taken up much of my thoughts.

I will be writing more about this (free) software in coming months - the problem that HumanEdj solves, namely how to collaborate more efficiently, appears to be pretty much ubiquitous in the working world. Things have certainly changed from when I first started publishing on the subject of Human Interaction Management, in late 2004! Back then, the claim that we needed a new form of process management for human-driven processes was treated with a mixture of incomprehension, disbelief and (to my surprise) even anger, from people who had spent a long time getting to grips with BPM as it was and mostly still is. Now everyone from Bill Gates to the BBC is talking about how the modern workplace is in crisis.

I hear this everywhere I go now, whether it is on consultancy assignments or chatting to people socially. The key symptom is poor use of email - a groaning inbox, being unable to get results from sending messages, misunderstandings between colleagues, and much more. A colleague recently pointed me at the interesting GoogleTech video Inbox Zero - if you don't have time or patience for the entire hour's talk, a summary is also available.

The talk is interesting not so much for its content, which as the presenter states up front is hardly rocket science, but for the number of hits - 147287, last time I looked! The fact that such a conventional chat about email handling is receiving this level of attention indicates how many people are struggling to make use of this most basic workplace tool, and how serious a problem it is. The Inbox Zero video is part of a sudden flood of talk on the topic, such as this article from the Wall Street Journal.

However, most of these writings simply scratch the surface of the real issues. As the summary of Inbox Zero referred to above points out:

Email is a medium. It’s not where the action is. I have a question here though: I agree with the fact that the action is not in email. But could we put the action there? If email is the core tool of a knowledge worker, shouldn't more tools integrate with email? E.g. why can't I just write a document in my email? Or the other way around: why can't I just write a document in Word and decide that I want to send the content as an email? (Note: I don't mean attaching the document to an email!)

Indeed. Further, there are deep underlying problems not addressed by any of these writers. Just to give one example, most people now routinely confuse asynchronous and synchronous communication - in particular they tend to use email where telephone is more appropriate. Hence emails acquire false urgency. There are many more examples, relating to deeper interpersonal issues such as responsibility, authority, confidentiality and competency.

The next version of HumanEdj, due for release later this year, tackles these issues head on. With zero configuration (if you use Outlook - otherwise you need to tell it where your email lives) it will categorise all your email for you, outgoing as well as ingoing, including both body text and document attachments. You'll then be able to see at a glance who you are currently dealing with, what the dealings are about, who you are waiting on, who is waiting on you, and much more - in particular, why all this is the way it is.

These features, however, are just the icing on the cake - you get them "for free", before you even start thinking about structuring your work in process terms, which is the real power of the software. However, it is the Human Interaction Management underpinning that makes the approach possible.

TAKE AWAY

Can you see the writing on the wall? It is a clear warning - if you don't act in the very near future to solve the human collaboration problems of your business, you won't have a business for much longer.

In the next postings to this blog, I will say more about these problems, and how they are addressed by the principles and patterns of Human Interaction Management. I also hope to include some sneak previews of the free HumanEdj software that provides full support for the ideas.

If you would like to be part of the "new world of work", rather than left behind by it, stay tuned.

Posted by keithhb in Management | Permalink | Comments (2)

Abraham Maslow, creator of the hierarchy of human needs, famously said: "If the only tool you have is a hammer, you tend to see every problem as a nail." But what if you have a Swiss Army Knife?

This is the 12th and final article in my blog series on The Future of Programming. The series started back in March, and has addressed various topics including:

• Dealing with acronym hell


• BPM without a BPMS


• Managing outsourced application development


• Professional development for developers


• Quality planning


• Managing change in large-scale development

In this concluding article, I will discuss a theme that underlies all these topics.

When I entered the IT industry, I was trained to become not just a software developer, but a software engineer. What is the difference?

Engineering is fundamentally about techniques and tools. An engineering job consists essentially of assessing the situation, deciding what techniques and tools are suitable, then applying them. Hence, in order to be an engineer, you need not only depth of knowledge, but also breadth - a wide-ranging understanding of problem types and solution types.

I began my IT career in the late 1980s. At this time, object-oriented programming was just getting started. Some would say that it is still is! However, there were certainly fewer mainstream programming languages back then, fewer (and less powerful) development tools, and far fewer programmers. Also there were hardly any packaged applications for business use. As a result, it took a lot longer and cost a lot more to introduce new software into an organization.

As a result, it was more common for large-scale software projects to be conducted on quite a formal basis. Not only was the choice of tools and techniques more limited, making it easier to apply an engineering approach to software development, but also the stakes were higher. Despite this, many projects failed, of course, but at least there was recognition - fueled by the writings of thinkers such as Fred Brooks - that software development needed very careful handling if it was to succeed.

Now, in today's world, we have a plethora of new development tools - including BPM suites claiming to make it possible for business users to build their own enterprise software, and SOA suites claiming to make it possible for an organization of any size to share bits and pieces of code willy-nilly. As a result, many organizations are dispensing with the costly and time-consuming engineering processes, in favour of what appear to be the latest and greatest quick-fix solutions.

Well, why not, you may ask? I have 2 main fears about current development trends.

First, organizations seem to be confusing tactical and strategic solutions. A set of BPMN flowcharts may speed up your logistics process, or shave 1% from your invoicing costs. But this is no substitute for an organizational model, a model that is based on high-level business goals. It is easy to adopt short term point solutions that in the medium term fragment your enterprise IT architecture, and in the long term render it unmanageable. In particular, the true problems of SOA Governance are only just starting to emerge. BPM and SOA may in fact be driving a return to the integration nightmare that they promise to solve.

Second, whatever happened to testing? I recently heard a BPM Suite vendor speak proudly of a BPMN process they had implemented that contained 250,000 steps. This represents a vastly complicated piece of business software, so I asked how they had tested it. He replied that testing was unnecessary since their tools did not require the user to write lines of code (just to draw diagrams), and anyway their products contained simulation features if you really wanted to prove that an application worked as expected. He then went on to say how this approach must be OK, since other organizations are using their suite to build safety-critical applications.

This response made me blanch, and still sends shivers down my spine. What sort of world is being constructed using such tools? Whether created via diagrams or as text, software needs testing - and simulation is no replacement for structured, automated, exhaustive, regressive verification. Further, anyone who knows anything about the theory and practice of simulation will recognize immediately that simulation of a flowchart of 250,000 steps is itself a hugely complex and risky exercise.

TAKE AWAY

No matter what tools you use, it is highly dangerous to take the engineering out of software development - yet this is exactly what is happening with the emergence of new BPM and SOA tools.

If you are responsible for software development, I suggest you ask yourself a simple question. Do I care what happens 12 months down the line?

If not, then you can probably ignore much of what I have written in this series.

Otherwise, I suggest you stop and take stock. Remember that much of what you read online and in magazines is written by people with a vested interest in getting you to adopt certain new commercial products. It takes only common sense to see that the practice of software development is nothing more and nothing less than engineering - and engineers in other fields are very cautious indeed about using new techniques and tools to build big things. Why should software be any different?

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

This is the eleventh in a blog series on the Future of Programming. Last time I started to explain the diagram of a "collaborative process pattern" that helps to bring large software projects under safe management control, by structuring the way you handle the continual revision to the development process that inevitably arises when different people are responsible for different aspects of the design. Please refer back to previous articles in this series for details. This time I will finish discussing (for now) how the diagram can help you to manage large software developments, by explaining the third and final perspective from which to understand it - the solution point of view - and how to put the diagram into practice as a means of managing ongoing operations.

From a solution perspective, the Role Activity Diagram is divided into two collaborative transactions - one at overall design level, and one at domain design level. Each transaction is bounded above and below by a 3-way interaction. In each case, the interaction at the top represents an exchange of requirements while the interaction below represents an exchange of solutions.

Both the requirements exchanged in the top interaction and the solutions exchanged in the bottom interaction will involve complex, compound objects. A solution interaction, for example, may pass across:

• The final problem statement, now fully developed
• The objects that form the solution, contributed by different Roles
• The models and tests that justify the solution
• A map of the dependencies between all these objects (for use in tracking the impact of changes) and a related approval chart, showing who has approved what. (Both these can be maintained automatically. Whenever a participating Role submits a new version of one of their deliverables, all the approvals that are dependent on the new deliverable are removed and new approval activities are triggered. Processes and object relationships interact - as long as they are both represented.)

In between the upper and lower interactions, each participating Role will carry out various different activities, in order to provide its own part of the solution. The Role Activity Diagram shown here is simplified and omits these activities, as well as the phases into which the activities will be broken during the sub-project. Why?

All we need at this stage is to represent a consensus from the initial meeting on how to proceed with the sub-project. We are not seeking to prescribe exactly what everyone will do to meet the requirements placed on them - just to establish a common ground on how the issue as a whole will be dealt with. What we are interested in initially is to make it clear to all parties what their responsibilities are to the process as a whole. We need to show only who is to do what, how they will deliver their outputs, and what impact each person's work has on the rest of the process.  The diagram effectively models not only the work required, but also the executive and management controls required over the sub-project.

Once the work gets going, it is inevitable that changes will occur to the process shown in the Role Activity Diagram. In particular, the activities in the sub-project will be extended, and broken into distinct phases, and during the course of the sub-project the elements of the diagram will be extended further as required. The project participants will make agreement depictions at overview or detailed levels that not only refine the process, but also allow all concerned to share the consensus reached about next steps.  In particular, the deliverables may change - perhaps certain ones will be altered to include only partial functionality, and the deficit remedied by other means (or just agreed with the customer as a concession against requirements).

It is here that day-to-day management control can be applied, in order to ensure that the process as it unfolds in operation takes place safely, in accordance with any applicable regulations, and with all parties committed to the same manner of proceeding.

Moreover, it will be possible to do this at either level without contravening the executive control applied from above, since it is clear from the diagram how this control is applied - via the application of the REACT pattern. Should the process participants at any time decide that this application was inappropriate, this change can be described in a simple way via a new Role Activity Diagram, and submitted to the appropriate Role for approval.  For instance, suppose that in a particular domain - safety engineering, say - that the Domain Design Authority feels it sensible to share some of his or her Research activity with the Designers of their part of the solution. After discussion with the Domain Sponsor and Designer(s) in question, the Role Activity Diagram can be altered to show the proposed process change, and sent to the Domain Sponsor for official approval.

There is a lot more one can say about such changes, but for now all we need to do is note that the diagram provides a simple means of achieving shared understanding on a new way forward, and of ensuring that this way forward is safely managed.

TAKE AWAY

Perhaps the primary concern for anyone charged with a large-scale software development should be management of faults, issues and other things that in some way force change to the current plan.  It is a law of nature that such changes happen during software development.  With small-scale software development, agile techniques suffice.  With large-scale software development, my experience, both first- and second-hand, is that they don't.  Having separate teams and/or sub-projects operating concurrently changes the situation fundamentally, and you need something more - something industrial strength - in order to cope.

In the last few articles in this series I have outlined a technique that can help with large-scale software development.  In the next - and concluding - article in this series, I will offer some observations on general emerging trends in enterprise software development.  In particular, I will discuss how the adoption of new BPM/SOA tools is bringing with it some extremely dangerous practices.  To find out what they are, and how to avoid them, tune in next time.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

To understand the pattern fully, it is necessary to interpret the diagram from three different perspectives:

1. Management
2. Participant
3. Solution

Let's start with the management perspective. This is based on the Human Interaction Management (HIM) principle known as separation of control.  Part of this concept is to divide management responsibilities into executive control (setting goals and responsibilities) and management control (monitoring and supporting the work itself).  Executive control comes from above - management control comes from within.  We also need the HIM concept of REACT, an acronym describing the breakdown of any work activity into 5 stages:

1. Research - find out what you need about the domain;
2. Evaluate - apply this information to the work in hand;
3. Analyze - decide from the options available;
4. Constrain - plan the work;
5. Task - do the work.

There is a lot more to separation of control and REACT than suggested here, but that will do for now.  Applying these ideas to the diagram:

• At the overall level, the Overall Sponsor is the Sponsor Role embodying executive control, while the Domain Sponsor and Overall Design Authority together embody management control. The Overall Design Authority has the Research, Evaluate and Analyze activities, while the Domain Sponsor has the Constrain activity. The Task work is not shown as a single activity, but split between Domain Sponsor and Overall Design Authority, and consists of work assignment and work validation respectively.
• At the domain level, the Domain Sponsor is the Sponsor Role embodying executive control, while the Domain Manager and Domain Design Authority together embody management control. The Domain Design Authority has the Research, Evaluate and Analyze activities, while the Domain Manager has the Constrain activity. As above, the Task work is not shown as a single activity. In this case it is split between all three Roles Domain Manager, Designer and Domain Design Authority, and consists of work assignment, design work itself, and validation of that design work respectively.

As the process unfolds, the Role Activity Diagram will be refined to include sub-components of each high-level activity depicted at the initial stage.  This can be done without contravening the controls established at the start - the pattern is fractal, i.e., it can be applied at a more and more detailed level without change to its structure.

The second way to interpret the diagram is from a participant's perspective. There are three Roles at the overall design level, and another three at the level of each specific design domain or discipline (database design, user interface design, security engineering, and so on). There is a degree of correspondence between these levels that indicates that the model is natural:

Overall design level:

1. Overall Sponsor: Acknowledges that the problem is real and kicks off the solution process
2. Domain Sponsor: At the request of the Overall Sponsor, initiates work in each of the design domains, coordinates the separate streams of work, and reports on progress
3. Overall Design Authority: Responsible for the integrity and quality of the whole design (this Role gives permission for the integration of the specific solution)

Domain design level:

1. Domain Manager: At the request of the Domain Sponsor, assigns expert resource to the problem as it applies to a specific domain, then facilitates and supports this work
2. Designer: Participates in the collaborative activity to develop a domain-specific component of the solution
3. Domain Design Authority: Responsible for the domain-specific aspects of the integrity and quality of the design (this Role gives permission for the domain-specific components of the solution to be released for integration into the overall design)

Different Roles, such as 1 and 3 at each level, may well be performed by the same person. However, since Role Activity Diagrams do not include Users, this is not shown explicitly. What we gain from the Role Activity Diagram with respect to process participants is an overall context for each person's work. This is done in a simple and effective way by the Role Activity Diagram, from which it is immediately obvious how each person's work relates to that of the others, who communicates with whom, and who is dependent on whom.

For instance, it is clear that a domain solution cannot be submitted until each component of that solution has not only been prepared, but has also been "Declared Complete" by the corresponding Domain Design Authority. Further, it is not even possible to submit a domain solution component that has not been approved. Submission can only be done in tandem with the Domain Design Authority, as part of a single interaction that also involves approval - a review meeting, perhaps.

TAKE AWAY

Complex problems - such as large-scale software development - do not have to have complex solutions.  The way forward is to apply abstraction: remove distracting detail, and generalize the situation, so as to arrive at clear and simple patterns underpinning all such situations.

The pattern shown in the Role Activity Diagram is such an abstraction.  In HIM terms, it is a collaborative transaction.

Next time I will explain the third and final way to read the Role Activity Diagram - from a solution perspective - and show how to get started with applying these principles in order to safely manage the development of your own software solutions.  No matter how large and complex your development process appears to be, there is always a simple way to manage it safely.  You just need to look at things from the right angle.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

The trick is to recognize the reality of how people work together - and use that as the basis for a structured approach to managing their activities.

Let's take a typical scenario. A design team is in the middle of a large concurrent design project, and an unexpected issue has arisen - sufficiently serious to warrant specific management attention, and potentially impacting several design areas (database, user interface, security, and so on).

Meeting to discuss the problem, team members realize that a new sub-project is required to manage the issue. Initially they have no plan or structure for this new sub-project. The input to the sub-project is itself a problem definition that may require further investigation and development. However, there are a number of significant things that they can say from the start. They may have none of the detailed elements of a process, but they are able to recognize a pattern. They know how collaborative problem solving generally takes place.

The pattern they recognize for collaborative problem solution has specific aspects that allow us to model it, at a high level, as a process:

1. The people involved will perform certain Roles, each with their own goals and responsibilities. There will be a number of participants in the solution process, with skills in different domains - database design, user interface design, security engineering, and so on. For each type of contributor, there may well be distinct phases of activity: problem definition, concept design, detailed design and testing, integration and testing, and so on. Hence, the sub-project is in fact composed of multiple distinct streams of activity, one for each domain. The contributions of all these have to be coordinated.
2. The eventual deliverables will meet certain preconditions. If successful, the output of the sub-project will be a solution, probably consisting of a set of solution components in different domains. The solution components in each domain will include models and tests that demonstrate that the solution solves the defined problem. There may well be interface specifications and other supporting documentation. Among all these objects there will be varied dependency relationships. Hence, the components of the solution both at domain level and overall are an interdependent set that must be protected from further change unless the whole solution is revisited and retested. The set of solution components can be viewed as a transaction - somewhat like a database transaction, although subject to quite different rules in which all sub-project participants commit jointly to the resolution of the issue.
3. The process participants will interact in a way to ensure that their individual contributions combine to produce overall deliverables of the highest possible quality. The components of each domain solution must be made available to an appropriate person for integration into a unified domain solution. Similarly, the domain solutions must be made available to an appropriate person for integration into a unified overall solution. The individual domain solutions and the overall solution must all be validated and certified by an appropriate authority, and this activity must be linked to the aforementioned integration.

With this analysis in mind, we can use a Role Activity Diagram to represent the sub-project as a whole:

TAKE AWAY

You may be reading this because you are responsible for a large-scale software development. And you may be wondering how the diagram above is going to help you. Is it the magic bullet you need? If so, why?

If you do not find the diagram above self-explanatory, this is because it draws on concepts from Human Interaction Management. Before you can understand the diagram, you need to understand the concepts.

In the next instalment of this series, I will explain 3 different perspectives from which the diagram can be viewed:

1. Management
2. Participant
3. Solution

Armed with this understanding we will start to see how even the thorniest - and scariest - problems in large-scale software development can be tamed. Complex problems can be broken down into a set of simple ones, so that management controls can be applied and the required solutions delivered.

If this sounds like what you need, tune in next time - and if you can't wait, you can always get the book :-)

Posted by keithhb in Management • Security • Service-Orientated Architecture | Permalink | Comments (0)

In the last posting to this blog, I wrote that most change management procedures as documented in a Quality Plan give "absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design."

To deal properly with concurrent design, one must somehow support the unpredictable impact across the entire project of design changes originating from one particular team.  In other words, one must stop being scared of change - trying to prevent it, deny it, or limit it.   Shift happens, and success means accepting that change is a natural part of the software development process.  One must institutionalize continual decision-making on what to do next.

This will break down badly if such decisions are imposed from above, since engineering is a highly complex discipline.  Put simply, decisions that do not involve those working at the coal face will usually be wrong.  So all concerned need to share in the decision-making process.

Agile methods have their root in this understanding.  However, talk to enough software project managers and you will find that agile projects tend to run into trouble when they are over a certain size.  It is hard to define the threshold precisely, but from experience I would put it at somewhere around 30 developers.  Why?

Here is a quote from Martin Fowler, generally accepted as one of the key authorities on Agile methods, taken from his seminal article on Continuous Integration

One of the features of version control systems is that they allow you to create multiple branches, to handle different streams of development. This is a useful, nay essential, feature - but it's frequently overused and gets people into trouble. Keep your use of branches to a minimum. In particular have a mainline: a single branch of the project currently under development. Pretty much everyone should work off this mainline most of the time. (Reasonable branches are bug fixes of prior production releases and temporary experiments.)

Here is the source of the problem in a nutshell.  Most agile practices assume that the mainline is shared between all developers.  This works fine for small projects, but is totally inappropriate for larger projects, where concurrent design is employed and as a result the work is effectively structured as a set of inter-related sub-projects.  Such sub-projects may diverge widely at times - some may never make it into the mainline at all, while others may only ever contribute part of their own output to the final deliverables.

A good comparison is with school class sizes.  Teachers tend to struggle when class size grows beyond 30.  Well, so do software project managers.  In both cases, the solution is to divide people into groups that are semi-independent, and can work at their own speed, making their own choice of subject matter and how far to pursue it.

In my view, there are 2 reasons why the software development community has not properly taken this on board:

1. Failure to understand configuration management techniques.  This is not really excusable, given that there are simple guidelines that anyone can follow to make use of multiple concurrent branches entirely safe.  Laura Wingerd of Perforce Software gives a great explanation of how to manage what she calls the Flow of Change: How Software Evolves (book excerpt) and presentation on same material.  The techniques she explains can be used with any modern software code repository, such as CVS or Subversion.
2. Failure to manage continual decision-making on what to do next.  This is more excusable, since patterns for such collaborative work have only recently emerged, in particular with the discipline of Human Interaction Management.

TAKE AWAY

Are you responsible for (or just working on) a large software project?  If so, you need to think seriously about the consequences of concurrent design.  Here is what IBM has to say (Best practices for software development projects, 10 August 2006):

Most software projects fail. In fact, the Standish group reports that over 80% of projects are unsuccessful either because they are over budget, late, missing function, or a combination. Moreover, 30% of software projects are so poorly executed that they are canceled before completion.

It makes no difference what you are building, or how - whether you are automating business processes using Eclipse MDA, automating business processes via a BPM Suite, or developing a stock trading system.  The IBM article above goes on to list as the first practice crucial to success:

It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.

Exactly.  However, you won't deal properly with the continual changes arising from concurrent design by trying to impose a rigid set of procedures.  What you need are organizational patterns naturally suited to a dynamic, evolving work environment.  In the next instalments of this series I will say more about such patterns, and how to implement them.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (1)

Here is a typical description of a Quality Plan, selected at random from the Web:

Quality Plan Description

The Quality Plan describes how a developer's overall quality process guidelines will be applied to a project. It defines what is meant by the various quality-related tasks in the Project Plan.

For example, a developer's quality manual may describe a review process for ensuring that delivered software meets requirements. The Quality Plan for the project tailors this general definition to the project at hand, specifying items such as who generates the requirements, what form these will take, who reviews them, etc. Thus, when a task in the Project Plan reads "Review and update requirements," the Quality Plan defines what will be done, and who will be responsible.

Quality Plan Contents

The Quality Plan outlines how you will build quality into the software and documentation. The dates assigned to key tasks in the Quality Plan are entered into the project plan.

The Quality Plan describes:

• How you control changes.
• How you ensure that the product meets the requirements (validation).
• How you ensure that the product works properly (verification).
• How you track multiple development builds of the software to avoid confusion (configuration management).
• How you plan for and execute testing, both incrementally during development and for the entire product before delivery.
• How you track and resolve defects.
• How and when you conduct design reviews, code reviews, walk throughs, reviews of test scripts, reviews of test results (for example, is 100% of all code checked, or only the most complex parts?).
• Definitions, methods, and criteria you use to determine whether the software has passed each review.

At first glance, this seems OK.  However, it is in fact rather a limited approach to quality planning.  The problem with such a definition is that one could get away with almost anything (or nothing) in the Quality Plan document, since the only expectation is to have some form of test and review during the life of the project.  Quality in its classical definition means "fitness for purpose".  So the real question that should be asked of a Quality Plan is this: how does it ensure that the software delivered is fit for its purpose?

A more thoughtful approach to quality planning is that of the PRINCE 2 method developed by UK government, and now an internationally recognized approach.  Here is a typical PRINCE 2 approach to quality planning:

Purpose

The purpose of the Project Quality Plan is to define how the supplier intends to deliver products that meet the customer's quality expectations and the supplier's quality standards:

• Does the plan clearly define ways in which the customer's quality expectations will be met?
• Are the defined ways sufficient to achieve the required quality?
• Are responsibilities for quality defined up to a level that is independent of the project and project manager?
• Does the plan conform to the corporate quality policy?

Composition

The Project Quality Plan should contain:

• Customer quality expectations
• Acceptance criteria, a prioritised list of criteria for the final product(s) that must be met before the customer will accept the final product(s).
• Quality responsibilities, who is responsible for each of the aspect of quality of the final product(s)
• Reference to any standards that need to be met
• The quality-control and audit processes to be applied to project management
• Quality-control and audit process requirements for specialist work
• Change management procedures
• Configuration management plan
• Any tools to be used to ensure quality.

Derivation

• Customers quality expectations and requirements
• Organisational or programme quality management system and standards
• Configuration management and change control requirements

Such a description will produce a more useful (if longer) document.  However, there is still a fundamental problem with it.  The problem lies in the innocuous phrase "Change management procedures".  Typically such procedures comprise some combination of:

• Techniques for issue tracking
• Guidelines for approval of proposed changes.

This gives absolutely no indication of the complexity inherent in managing the sort of changes typically encountered during the life of even a moderately complex software project.  This is because most enterprise software projects rely on concurrent design.

Demands for increased productivity and shorter times-to-market generally force software engineering organizations to adopt concurrent design.  Different parts of a complex application are developed in parallel and then integrated, shortening project schedules. Typical interactions in this type of business process are activities such as agreeing on sub-contract terms, signing off a specification, concluding a project stage, and so on. Each of these may involve several parties, contain processing spread over varied systems on different platforms owned by different organizations, and require a number of steps to complete successfully. Further, although efficiency gains can be achieved via this approach, it carries risks of rework, which arises when concurrently performed work turns out to be incompatible, and an impact of one interaction is that others need to be repeated.

In other words, concurrent design is what Human Interaction Management defines as a typical human-driven process - fraught with instability, since software engineering managers are dealing with activities that are only partly predictable when the process commences. It is a fundamental characteristic of the whole environment that it is organic and dynamic, so hard coded process descriptions are unsuitable. Even if they are supported by tools that make modification of that process possible, the participants are innovative, creative people who need the flexibility to put their skills and experience to use in what seems like the most appropriate way at the time.

There is a parallel situation in the structure of the product itself. A complex software application may appear to decompose neatly into components (and services) but in fact it is a densely tangled set of solutions to a huge and varied set of requirements and constraints. Later modifications only increase the complexity. New connections and dependencies are introduced by tracing the impact of modifications, defining and implementing consequent changes, testing the whole package and restoring the integrity of the product.

TAKE AWAY

Software product managers attempting to deal with applications built using concurrent design generally find themselves tearing their hair out by the time the project is halfway through.  If they are lucky they will be running to stand still.  If not, they will find the project slipping further and further every week.  Failure to properly handle concurrent design is probably the single biggest cause of software project failure.

The solution to managing concurrent design lies in giving the ever-changing nature of a human-driven process the respect it deserves.  In such work, a major part of the workers' duties is to decide on next steps.  This must be done both continually and collaboratively, since there are always new issues to resolve and few of them can be dealt with in isolation.

In the next instalment of this series, I will look at techniques for structuring this "continual decision on next steps".  If you want to make your change management as elegant and efficient as your software code, stay tuned.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (0)

My last entry generated enquiries by email.  I mentioned that "In my own consultancy work, I provide clients proposing to engage in an outsourcing project with a learning trail that their key technical staff should follow, customized to the particular type of project they are considering" and some people asked for examples.  So here is such a learning trail, for Eclipse Plug-in Development using Java.

Note that such a trail can be used both by developers employed by the outsourcing supplier and by those people in your own organization with technical responsibility for the projects concerned.  Doing so will ensure that all the people involved have a proper understanding of the technologies and approaches required.  This is the first step in establishing the kind of close and mutually supportive working relationship that is necessary if the projects are to succeed.

The second step is to decide what outputs you expect from each project.  This will be a lot more than a set of software code modules, since you also need everything necessary for maintaining the solution(s).  Most projects take care to produce the various forms of application documentation required by new users, regular users, power users, system administrators, support staff, operations staff, and so on.  They may also have a requirements document and/or some form of system specification.  However, there are some additional outputs that often get paid only lip service, yet are crucial to success:

• System Architecture.  While software documentation is increasingly being integrated into the code itself, as well as into the static/dynamic models used to generate the code, some high-level information should always be kept separate, and its ongoing maintenance treated as a priority.  This information includes a description of the system's purpose and how it achieves it, the decisions made about key implementation issues such as concurrent use, deployment considerations, licensing strategy, placeholders for future enhancement, and so on.
• Technical plan.  It is necessary to consider up-front the techniques to be used for developing the software, and choose suitable tools with which to implement them.  Such techniques range from high-level work management practices, for example the many variants of agile methodologies, to low-level code development matters such as configuration management strategy.
• Quality plan.  How will you ensure that (every release of) the application meets user needs?  Typically, a combination of strategies is necessary, ranging from code-level practices such as test-driven design and continuous integration through to quality assurance practices based on the principles of Human Interaction Management.

Let's note some things about these documents.

First, such documents are required whether you are implementing business processes via a BPMS, implementing business processes via Model-Driven Architecture, or implementing anything else at all.  Whether you are building a stock trading system for a bank or an embedded device driver for a mobile phone, software construction has certain general principles that must be adhered to.  The rise of agile thinking, and the associated rise of "New Methodologies" designed to handle requirements changes on the fly, have led many people to abandon tried-and-tested software construction practices as old-fashioned.  Beware of throwing the baby out with the bathwater!

Second, the documents do not have to be long.  Rather, the shorter the better.  As long as they cover the required ground, and are consistent with your actual working practice, 3 pages are always preferable to 30.

Third, if you are not going to keep these high-level documents up-to-date, you may as well not bother creating them at all.  Anyone who has ever developed any software at all knows that it always seems to be a moving target.  Everything changes under your feet from the very first day.  In the white heat of a development project, the extra effort required to maintain high-level documents such as those described above may seem like the last thing you should be worrying about.  But actually it is almost the most important thing you can do.

TAKE AWAY

The high-level software project documents described above are  not only your personal bomb-proofing - the evidence, if things go wrong, that you personally acted as a responsible professional - but also the bomb-proofing for the project as a whole.  Armed with current, complete high-level documentation it is possible to salvage many a software project that appears doomed - certainly such documents make it far easier to hand the work over to a new outsourcing supplier, or take it back in house.  Without them, you are working blind.

My first step when engaged to rescue an ailing software project is to make sure that the documents described above are in place and up-to-date - creating them from scratch if necessary, as a retrospective exercise.  Before you can make decisions about where to go next, you need to know where you are now.

In the next postings to this blog I will say more about how to guarantee the success of a software development - and show, in particular, how the often-overlooked Quality Plan can be used to make life easier for everyone.

Posted by keithhb in Business Process Management • Management • Service-Orientated Architecture | Permalink | Comments (0)

To do this, you must at a minimum be able to understand the work that is being carried out.  More accurately, one needs not only to understand the work, but further, to understand how to do the work well.  Then you can step in if it is not being done in the most appropriate manner.

So here is the first step towards outsourcing success: make sure that your own organization has a professional development capability.  Even if this consists only of a couple of individuals, it is essential to have people in-house who are genuinely capable of judging the deliverables provided by an outsourcing company, and positively influencing the way in which the work is being carried out.

Unfortunately, it is getting very hard nowadays to see the wood for the trees in enterprise IT.  A recent Computer Science graduate, for instance, is likely to lack the key knowledge and skills required to make the most of an increasingly complex development landscape.  As I described in the first post in this series, the current profusion of acronyms is enough to bewilder anyone, and it's getting worse daily.

So it is useful to step back from the sea of neologisms, and divide the sorts of skills required into levels that are independent of any particular technology or methodology.  This can be done as follows.  Here is a graded series of skill levels that a software developer should rise through on their way to becoming a modern professional:

1. Language.  This is what someone will typically know coming out of University, or after having taken certifications in a programming language.  It consists of knowing the core library classes/functions of the language(s) concerned, as well as techniques for exception handling, reflection, concurrency, and so on.  Note that superficial knowledge is not enough here - to write a language well, or code review other people's work, you need to know and understand all the features that are available out of the box.
2. Language skills.  Once you know a language, you must then learn how to use it.  This involves stylistic aspects - the way you structure, format and document code. It also involves technical aspects - how to use concurrency safely, pessimistic rather than optimistic programming, how to instrument code to permit analysis/debugging, the different forms of collection object and their uses, proper management of loading/binding, and so on.
3. Design patterns.  Since the late 1980s it has become accepted practice to utilize standard abstraction techniques when writing code, mainly for maintainability but also for code quality and productivity.  The key reference here is the "Gang of Four" book ("Design Patterns: Elements of Reusable Object-Oriented Software" by Gamma, Helm, Johnson and Vlissides), which uses C++ and Smalltalk for its examples.  The GoF book has inspired a decade of further research. A professional programmer must know not only the 23 design patterns from the GoF book but also many additions to and enhancements of these patterns. They will also understand when and how to use each pattern, and how to refactor code - to restructure it to conform more (or less) closely to specific patterns, or just to improve its elegance and readability, without changing its functionality.  As well as for programming, there are also design patterns for enterprise application architecture - the key reference here being Martin Fowler's book, "Patterns of Enterprise Application Architecture".
4. Frameworks.  No-one now writes enterprise applications without using an abundance of frameworks - code libraries that embody best-of-breed solutions to common technical problems. Frameworks are based on design patterns, and you cannot properly understand how to use frameworks unless you first gain confidence with application of design patterns. Many frameworks are open source and can be used to build a product intended for commercial sale.  Frameworks often have associated domain-specific languages such as XMI (XML Metadata Interchange) and OCL (Object Constraint Language).
5. Standards. Any software going into a corporate environment must interact with other software in a standardized way.  Hence such applications need to conform not only to accepted standards but also to emerging standards. For instance, any Java-based application that talks to a content management system must be aware of JSR-170 and JSR-283, which are standards for communicating with such a system via the Java language. There are many important standards pertaining to application development, some like the above arising from a language-specific community, and others (such as Topic Maps for XML documentation of linked content, or DocBook for technical documentation) from non-language-specific bodies such as ISO and W3C.
6. Work Management.  Any professional developer should have some grasp of the various possible approaches to managing a software project, ranging from traditional waterfall techniques through to the many different agile methods and techniques.  There are also design patterns for work management - the key references here are "Organizational Patterns of Agile Software Development" by Coplien and Harrison and my own book "Human Interactions".  I will have a lot more to say about this in future postings, since agile techniques are key to the successful management of outsourced application development - whether or not the project is officially viewed as being "agile".

TAKE AWAY

The first step towards successful outsourced application development is to make sure you know what is going on.  This means that your own staff must be able to understand technical matters at least as well as your outsourcing supplier.  To achieve this, you must ensure that your organization possesses people with skills at all the levels described above.  Some of these people will have only skills to a certain level - but you will need at least some people who have all the skills right up to level 6.   The number of such people required depends on the size of your organization and the number of outsourcing projects you have going on.

This is the first step towards outsourcing success and the most crucial.  In my own consultancy work, I provide clients proposing to engage in an outsourcing project with a learning trail that their key technical staff should follow, customized to the particular type of project they are considering.  If their people are familiar with the material already, all well and good.  If not, acquiring this knowledge not only prepares them for the project in question, but is a valuable contribution to their organization's general capability - a contribution that will stand the organization in good stead in future (as well as, of course, being useful professional development for the individuals in question).

Only once your organization has developed such an in-house capability are you in a position to code review the work that is done on your behalf, as well as to engage with your supplier as to how the work is being carried out.  In the next postings to this series I will discuss how to position yourself so that such engagement is not only painless, but viewed as beneficial by both sides.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (2)

In this article, I will discuss a change of which no-one in IT could fail to be aware - the rise of outsourced application development - and its impact on development practices.

Let's start with some typical problems that arise with outsourcing application development.  Here are some real outsourced projects on which I was brought in to consult:

• A medical system that passed all system tests first time - tests written by the developers - but that turned out to consist of unmaintainable spaghetti code;
• A telecomms data warehousing system that ended up completely dependent for maintenance on a single programmer who had written the core code, who became impossible to work with, and who then decided to retire early;
• An online catalogue with a hopelessly impractical database design in which individual queries often ran into thousands of lines - of SQL code, not results - and consequently each took hours to run;
• A retail replenishment system that had to be completely ditched and re-written from scratch in-house (under huge time pressure) due to the poor quality of delivered code.

These are not particularly exceptional cases.  As with anything in life, you get what you pay for.  Yes, you can save money by using staff from less developed economies - but unless you supervise the work properly, which means managing the code development as closely and carefully as if it was written in house - you will end up with only a short-term gain, and quite possibly not even that.

Why?  Are local developers (wherever "local" happens to be for you) always so much better than developers from other countries?

Of course not.  However, there are 2 key issues to be aware of.

First, programmers are human.  Suppose you are a programmer working under pressure for a company on the other side of the world, a company that does not know you personally, on a system that you will probably never see again once it has been delivered.  Being realistic, you are not going to polish every line of code until it gleams.  You are going to produce as much code as possible, as fast as possible, to the minimum acceptable level of quality imposed by your managers - or to a lower level, if you can get away with it.

Second, outsourcing companies cannot afford to encourage their development staff to invest time in personal development - in deepening and broadening their skills.  Such companies compete intensely on price.  They may be willing to pay more for senior developers, for whose services they can in turn charge more to clients, but they are generally unable to afford to sponsor the increase in knowledge and experience required to turn a junior programmer into such a person.  This means that high-level skills are acquired and applied piecemeal, if at all, by developers working in less developed economies.

As a result, there is an endemic culture of low quality code in outsourcing companies - a culture that means many apparently low-cost systems commissioned from outsourcing companies are in fact unmaintainable, or may not work properly even if they pass all their system tests.

TAKE AWAY

Does your organization rely on outsourced system development for mission-critical processing?  If so, the poor quality of such systems could end up as your undoing.

Coding standards are dropping across the IT industry - even where you would least expect it.  I have seen code written by senior developers with an outsourcing background as part of an extended interview process.  The purpose of writing this code was as an example to demonstrate their abilities, and hence to secure highly desirable work in the West.  Hence one assumes the authors invested as much time and trouble as they could into the example code.  However, while the code they produced did seem to work, the poor quality was obvious at a glance.  The code had been written with such disregard for conventions of style, documentation, etc that maintenance would have been a nightmare.

The authors of this code were all the cream of the crop - highly educated, with experience working on projects for companies such as Google and Amazon, and earning a very respectable amount already - typically around 40 USD/hour, in India!  The problem was that they had never been through the kind of induction process that used to be standard in every professional software development organization.  They did not know even what expectations to set of themselves.

In the next postings to this blog, I will discuss the nature of a professional development induction process, and how to re-introduce it in a world of outsourced application development.  I will also show how to ensure that code delivered from an outsourcing organization is of high quality.

There's nothing wrong with outsourcing application development - but you have to manage it the right way.  If you'd like to find out how, stay tuned to this blog.

Posted by keithhb in Management • Service-Orientated Architecture | Permalink | Comments (4)

• Analysts have a lucrative business in providing feature comparisons, advising vendors which new functions the market would be most interested in, and helping organizations select from the bewildering array of products on offer;
• Consultants have built a niche helping organizations choose, implement, and maintain these beasts - what a French consulting company with whom I once worked referred to disparagingly as "pousse-bouton" work;
• Enterprise architects charged with business process implementation are able to divest themselves of responsibility by laying it on the shoulders of software product vendors and their promises;
• Software product vendors have found a new market, often for a mixed bag of legacy or bought-in products that they can assemble, revamp, and re-purpose as a BPMS suite.

Nevertheless, it can't last forever.  The major vendors in IT are already owning up to the imminent death of the BPMS implicitly, not only by incorporating BPM functionality into monolithic middleware suites (Oracle, SAP, IBM, etc) but even into the operating system itself (Microsoft).  As I pointed out in my last post, published research from IBM shows how to implement business processes without a BPMS at all.  And here is what David Frankel of SAP has to say about using Model-Driven Architecture (MDA) for business process implementation on bptrends.com this month:

Model-Driven Service Orchestration

Often times, developers trying to get a handle on MDA approach it with the thought that they have
to use it to generate the components of their applications, or at least to generate fat skeletons of
the components, to which programmers add some finishing code. Then they run straight into the
last mile struggles with legacy back end systems that hold so much of the data in non-normalized
form.

Some of the most successful approaches that I’ve seen take a different route to nailing down the
last mile with MDA. They use service-oriented architecture (SOA) to present existing application
functionality as a set of service components whose interfaces are highly normalized via
consistent use of service interface languages such as Web Services Description Language
(WSDL). They use modeling to define specific orchestrations of the services. They walk the last
mile by compiling or executing the orchestration models.

In sum, this approach deals with the last mile problem that non-normalized back end systems
pose, by using SOA to provide a normalized intermediate tier that model compilers and model
execution engines can target in a consistent fashion.

In other words, Frankel is saying that you can use MDA to execute mechanistic business processes via SOA.  Draw your business process models, using whatever notation you like as long as it can generate code - UML is fine, for instance.   Then connect your code to services via WSDL.  That's it.  You can use a BPMS if you wish.  But you don't need to.

TAKE AWAY

Some people are attached to BPMS tools because of the wide range of features they typically offer - for messaging, reporting, configuration, and so on. However, tempting as such goodies may be, their use is not in any way related to increased business success.

In fact, there is a strong argument that adoption of multi-functional workplace tools has exactly the opposite effect.  A wide range of features and options often does little but distract people and muddy the waters.  Being bombarded by information from every side and in every form imaginable is a primary cause of stress - when you add multiple forms of messaging to multiple forms of reporting you get what I call "network overload", which is even worse than information overload.  On top of this many organizations are now using sophisticated workplace tools to transfer system administration responsibilities that properly belong to the IT department - authentication and authorization, for example - to end-users who are neither adequately qualified nor adequately resourced to handle such tasks.

What I see is that the net effect of introducing ever more highly powerful applications into the workplace has simply been to obscure individuals' true business goals and responsibilities.  How many people feel that they are running to stand still these days?  Or even just trying to keep their feet on the ground in the face of a tidal wave of information from all sides - and the demands for response generated by such information?

What the IT industry should be delivering is agile tools - agile in the sense of just-good-enough.  On this basis, the rise of programming techniques based on Model-Driven Architecture will, for many organizations, make a BPMS unnecessary.

For a start, an MDA approach is far closer to the original Third Wave ideal for BPM than the current generation of BPMS tools, since mainstream MDA tools such as Eclipse EMF are based on a simple, general, robust underpinning - unlike any current BPMS products!  It could be argued that the OMG MetaObject Facility that underpins EMF is the closest we currently have to the sort of generic framework envisaged by Smith and Fingar in their seminal 2003 book.

More importantly, MDA makes it possible to define your business goals, then implement processes to support them as quickly and simply as possible - and go round the loop again each time the goals change, as they do continually.  So why do you need anything else?  KISS, as they say.

Happy Easter.

Posted by keithhb in Business Process Management • Management • Office Applications • Security • Service-Orientated Architecture | Permalink | Comments (8)

... well, just about everything.

Some of you will atready know that HumanEdj went on general release yesterday.  The press release is here and you can listen to a podcast with Elizabeth Book about it here.

This release follows months of beta testing, in a programme that included over a hundred organizations of all sizes, types, sectors, and geographical locations.  I knew from my consulting experience that the software met a need in all sectors, so the variety in the programme was not really a surprise. What was a pleasant surprise was:

• The lev