Had a fascinating conversation yesterday with Jim Sinur, someone who understands very well what is coming in IT - namely, that the emphasis is shifting from server-side application automation to client-side human interaction.

In particular, dominating the desktop is going to mean something quite different in the 21st century. Whether or not Microsoft remains the leading operating system and office application supplier, the current trend towards commoditization of such products will eventually render Microsoft's presence more akin to that of a company like Cisco. Most people with an interest in IT know roughly what Cisco does, but most ordinary mortals don't think of themselves as Cisco customers.

So who or what will dominate the desktop in the 21st century? The increasing pressures on knowledge worker productivity that I often discuss in this blog means that we are all coming to expect more from workplace IT. In particular, we don't need cleverer tools - office applications, for example, have provided more than enough for most of us for a long time now. What we need is a way to join up the work we do using such tools.

Jim refers to this new layer of technology as the knowledge bus, which I think is an excellent term. It certainly describes very well what I have been trying to achieve with HumanEdj. Here are some key characteristics of a bus:

• A bus is a mechanism for crossing boundaries. A server-side business application is typically intended for use within a related set of organizations, since someone must own the servers on which it runs. Knowledge work is not like this, however. More often than not, it spans organizational boundaries - typically since both customers and suppliers are involved, but more generally due to the trend towards outsourcing and other forms of collaborative partnership that has come with globalization.


• A bus carries a payload. Transmission of information is the essential precursor to knowledge work, which can be viewed as the process of turning information into knowledge and decisions. This information can be structured (think Business Intelligence), unstructured (think emails), or semi-structured (think documents). A knowledge bus must be able to handle all these and more.


• A bus provides an infrastructure in which routing decisions can be made.. Here things get really interesting. How are routing decisions made in knowledge work? By humans, yes, but to support them and increase their efficiency the bus should allow the use of Business Rules in combination with human decision-making. This is vital for several reasons, not least to reduce the likelihood of uncontrolled rule evaluation leading to business disaster, such as is commonly thought to have happened in the UK Stock Market on "Black Monday" (19 October 1987).

TAKE AWAY

The latter aspect of a knowledge bus - routing - is in some ways particularly interesting.

Looking forward, it may be that we will see a convergence of Business Rule Management and technology support for knowledge work. If human intervention in rule evaluation is to be controlled properly, it must be viewed as a collaborative business process - what HumanEdj calls a Story. Hence, a precursor to safe use of Business Rules in an enterprise environment is the implementation of techniques and technologies for support of Stories.

Further, the creation and administration of Business Rules in the first place is just as much a candidate for control via Stories as their operational usage. Here again increasing competitive demands will force new levels of complexity in business operations, via extended Business Rule support - but without proper safeguards on the implementation of this support, we will only see more disasters like Black Monday,

20th century technology is not going to go away - if there is one lesson that the IT world should have learnt by now, it is the extraordinary longevity of legacy systems. However, it is certainly not going to be enough to meet the demands of the 21st century. The missing piece is the client-side, desktop-based infrastructure that allows organizations to leverage - safely - the enterprise backbone into which they have already invested so much money and effort. Thinking of this new infrastructure as forming the knowledge bus may be an illuminating way to make sense of what is coming.

Posted by keithhb in Knowledge Management | Permalink | Comments (0)

... well, just about everything.

Some of you will atready know that HumanEdj went on general release yesterday.  The press release is here and you can listen to a podcast with Elizabeth Book about it here.

This release follows months of beta testing, in a programme that included over a hundred organizations of all sizes, types, sectors, and geographical locations.  I knew from my consulting experience that the software met a need in all sectors, so the variety in the programme was not really a surprise. What was a pleasant surprise was:

• The level of response, since the only invitations to join the beta programme were mentions on this blog
• How many major incumbent software vendors were on the programme.

When I first started writing about Human Interaction Management (HIM), at the start of 2005, there was considerable resistance to the ideas from the software community.  Despite my efforts to explain that HIM is not competitive but complementary to all existing offerings, raising rather than diminishing the value of current software products, many people involved in the software market seem not to have believed this at all.  Rather, they saw the emergence of a "new breed of productivity software" as a threat.  This viewpoint has definitely changed - out of necessity.

We are currently reaching the peak of a hype curve - not only for existing middleware solutions (Business Process Management, Content Management, Business Rule Management, etc) but for the clutch of good-looking new tools known as Web 2.0 (wikis, blogs, AJAX, etc).  How much money are organizations actually making from adoption of any of this?  Very little, I would say.  And saying it, I often feel like the little boy in the story about the Emperor's new clothes - it can't be long before others start to say it too.

When the wave breaks, as it will soon, incumbents providing these products and services will either go under (i.e, watch consumers become disenchanted with the offerings into which so much money has been invested) or surf - by providing their user base with a step-change in how they extract value from such offerings.

Human nature being what it is, it will not be long before the people currently investing in new middleware start to ask how it differentiates them from their competitors - and those still in love with Web 2.0 start to reject reading blogs, editing wikis, and making customized charts in a browser as a waste of time.  To keep these customers, you must offer something more, something that speaks directly to their most immediate need - which is to succeed in whatever it is that they are currently trying to accomplish.

The business person's true needs are not for more information on a Web portal or more features in an ERP suite.  Rather, the business person needs to become more productive, both personally and on behalf of the employer on whose success they depend.  Hence the business person desperately needs a new type of software tools - tools that will help them achieve rather than just "do stuff".

TAKE AWAY

The coming change is actually an opportunity, not a threat, for a supplier of Web portals and ERP suites.  By providing goal-directed collaboration tools that are customized to integrate seamlessly with their existing offerings, such a supplier can show people how to leverage such offerings for direct and immediate advantage, thus increasing both adoption of these offerings and customer satisfaction.

This is why so many software vendors signed up to beta test HumanEdj.  It is a free goal-directed collaboration tool designed from the ground up to support such integration - it can be branded and extended via standard Eclipse plug-ins.  What software vendor wouldn't think it a good idea to offer something based on this to their customers, something that in helping them meet their own business goals incidentally brings them back to the vendor's own fold?

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications • Service-Orientated Architecture | Permalink | Comments (0)

Or rather, how not to lose money.

Happy New Year.  Regular readers of this blog may be waiting for the next instalment of the current series on SOA.  In the first two instalments:

• Is your SOAP turning to SOUP?
• SOA and the invisible side-effects of refactoring

... I outlined some of the business issues facing organizations implementing SOA, such as managing change and assessing cost effectiveness. It has been generally accepted for some time that investment in IT is not, on its own, going to give your organization any strategic advantage.  To the contrary, it is quite possible that investment in SOA will simply bring you a bunch of new expenses.

After all, what does SOA really promise to the business?  New integration technologies are just what they say on the tin - new technologies for integration.  Why should spending new money on integration improve your organization's bottom line?

I am not recommending that anyone should abandon their SOA initiatives.  Rather, I am suggesting that such initiatives should be driven by business people, with business aims in mind.  SOA proponents all claim that "yes, this is exactly what we do already".  But the "business advantages" espoused by the technical people currently heading up SOA projects are not usually business-related at all.  They are technical advantages, whose effect will only be felt by the IT department (if at all).

So what sort of things can be considered "business advantages"?  Is there anything generic that we can usefully say?  And how does it relate to SOA?

Let's start by recognizing that an organization is made up of people, all of whom have their own issues.  So "business advantages" means different things to different people.  Here is a very generic breakdown of the sorts of people involved in running any organization, and at an extremely high level, their most pressing problems.

The board of directors are there to lead the organization as a whole.  What they do is mandate aims (strategies and initiatives to implement) and constraints (policies and regulations to adhere to).  So the generic problem that board members have is very simple: how can they ensure that these aims are achieved and these constraints are complied with?  This is effectively a demand for "commitment processing".  The board wish to specify to executives how the business should operate, then receive from these executives their commitment to do so, plus (as time goes on) statistics on how things are progressing.

Managers have slightly different problems.  They have to deal with one-off ventures, projects, and issues.  They may be responsible for how the organization handles routine cases and problems.  Their fundamental need is to make sure that all these kinds of work are done efficiently - which means imposing enough structure for it to be facilitated, monitored and measured.  Some of the work can be (semi-)automated via workflow/BPM, but only some.  For the rest, such flowchart-based tools are far too restrictive.  However, the current alternatives - messaging and document sharing, perhaps via Web 2.0 technologies - don't provide any structure at all.

And then there's everybody else. The average office worker.  This is not a meaningless concept, since most people (including, in fact, directors and managers) have a basic problem in common.  Peter Drucker pointed out back in 1966 that "The Effective Executive" is good not at carrying out tasks, but at using time.  But these days, none of us have enough time - we are overloaded.  There is information overload: the well-known struggle to keep up with everything you are expected to know in order to do your job.  Even worse, there is network overload: characterized by the deluge of messages cluttering up the email inbox of the corporate employee.  The BBC say that:

It’s not unusual for office workers to spend as much as two hours a day, every day, sorting and reading all the mail which pours into their in-boxes, let alone the time they have to spend responding to it.

We all suffer from too much communication, which gets in the way of true collaboration.  And it is collaboration that delivers true business value.

TAKE AWAY

What does the analysis above of generic business problems have to do with SOA? 

Simply that we need to start by recognizing core business problems if we are to deliver genuinely useful business solutions.  In fact, effective enterprise IT architecture can help deliver solutions to all the problems outlined above - but only if the services provided by such IT are based directly on business needs, rather than technology features.

Most ERP system functions are never used.  Most "business intelligence" data is ignored.  Most "single sources of truth" bear little relation to the information truly needed by executives.  And the enterprise systems that people do use end up, as often as not, wasting people's time rather than saving it, like CRM systems that make you first do the work and then write it up.

In the second post from last year referred to above, I proposed that the answer to such business problems lies in Human Interaction Management (HIM), a set of principles, patterns and techniques for structuring collaborative work.  HIM is aimed directly at solving such problems, and is now fully supported by free software tools.

In the next posts to this blog, I will show how HIM can be used to leverage SOA to gain true business advantage.  In particular, we will see that HIM offers a direct way to understand change to services, and to measure their cost-effectiveness.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications • Service-Orientated Architecture | Permalink | Comments (1)

What?  The Chair of the Workflow Management Coalition, and former CTO (and designer) of Staffware, saying workflow sucks?  Yes, it's true.

Jon is not exactly noted for reluctance to rock boats.  But this statement seems particularly challenging.  I'll look more closely at Jon's message, and the content of the white paper he recently published, in a moment.  But first, we need to step back a bit.

Consider my post last week, How to abuse your software investments, in which I asserted that the deluge of new Internet tools known as Web 2.0 is actually a disadvantage to most organizations.  People are spending more and more time using these sexy new programs and devices without having any way of measuring the contribution to business goals made by such use.  As a result, most organizations are wasting more staff time than ever before.  Hardly the spirit of "extreme competition" needed in the new and challenging 21st century business environment!

To take just one example of such time wasting, here is a quote from the BBC, to which Jon himself drew my attention:

It’s not unusual for office workers to spend as much as two hours a day, every day, sorting and reading all the mail which pours into their in-boxes, let alone the time they have to spend responding to it.

How much is this contributing to personal or organizational goals?  I've written about the problems of corporate email before.  But email is just one example of how human work is actually being hijacked by new workplace technology.  Blogs, wikis, mailing lists, forums, intranets, chat, Web search, .... how are you measuring the operational (dis)advantages resulting from use of such tools in the workplace?

Here is a picture of where most organizations are now:

The underlying problem (and here we are getting closer to Jon's message) is that traditional work management tools - such as workflow - are of no help at all here.  You might think of workflow et al as "Work 1.0" and current knowledge-focused, collaboration-intensive, and innovation-driven work practices as "Work 2.0".  Here is a picture of where organizations need to go:

The key element of this picture is the blob in the middle, Human Interaction Management System, or HIMS.  What is an HIMS?  And why do we need one in order to implement what Jon's white paper calls Knowledge-Intensive BPM, or KIBPM - to control fundamentally important business activities such as Project Management and Case Management?

Let's start with the definition of an HIMS.  Here it is, from "Human Interactions: The Heart and Soul of Business Process Management" (2005), the seminal source book on Human Interaction Management (HIM):

A process modeling and enactment system that provides native support for the six Role Activity Theory object types (Role, Entity, Activity, User, State and Interaction), uses a state-based approach to Activity enablement and validation, permits Interactions to be composed of multiple asynchronous channels, and supports management of process change by allowing any process component to be created and configured as a natural part of process execution—not just objects of the six fundamental types, but also the user interfaces by which they are presented (screens, for example) and the means by which they interact with other systems (Web service calls, for example).

This is a complex definition, of course - almost what you would expect to see in a software specification document.  This is deliberate and necessary.  A HIMS must be a very specific sort of beast, in order to support the 5 basic principles of HIM:

1. Connection visibility - to work with people, you need to know who they are, what they can do, and what their responsibilities are as opposed to yours.
   You need Role and User objects, both instances and types, each with its own properties and responsibilities.
   
2. Structured messaging - if people are to manage their interactions with others better, their communications must be structured and goal-directed.
   You need Interaction objects in which there are multiple asynchronous channels, each for a different purpose.
   
3. Support for mental work - organizations must learn to manage the time and mental effort their staff invest in researching, comparing, considering, deciding, and generally turning information into knowledge and ideas.
   You need Entity objects that can be created, versioned and shared within a process.
   
4. Supportive rather than prescriptive activity management - humans do not sequence their activities in the manner of a procedural computer program.  There is always structure to human work, sometimes less and sometime more, but it is not the same kind of structure that you get in a flowchart.
   You need State objects that can both enable and validate Activity objects, along with the Roles that contain them.
   
5. Processes change processes - human activities are concerned often with solving problems, or making something happen. Such activities routinely start in the same fashion - by establishing a way of proceeding. Before you can design your new widget, or develop your marketing plan, you need to work out how you are going to do so - which methodology to use, which tools are required, which people should be consulted, and so on. In other words, process definition is an intrinsic part of the process itself.  Further, this is not a one-time thing - it happens continually throughout the life of the process.
   You must be able to manipulate not only objects but also user interfaces and integration mechanisms via the process that contains them.

How could a conventional "task allocation and notification" system possibly provide all these vital features?  Just to take one example, I have never seen a workflow/BPM system in which it is practical for users to try and make significant change to a process from within the process itself.  This is why Jon Pyke says, of course with tongue in cheek, that workflow sucks.  We need a new form of software tool, in order to support the business practices that are at the heart of organizational efficiency in the 21st century.

TAKE AWAY

 If you want to compete in the 21st century, you need to leverage your human resources efficiently.  If you want to leverage your human resources efficiently, you need an HIMS - not a workflow/BPM system rebadged as an HIMS.

So ask any software vendor trying to sell you a "Human Interaction Management System" how their offering conforms to the definition above.

Don't let them sell you a pig in a poke.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications | Permalink | Comments (3)

For a start, there are multiple means by which I communicate with each beta tester - typically a combination of telephone, Skype talk, Skype chat, and email.  So the individual communications end up scattered about in various different places: my head, scribbled notes, Skype, and Thunderbird.  I try to record all interactions in a single repository, but this always means first having the conversation then writing it up (or pasting it in).  So inevitably some pieces of each conversation get lost, or collated out of sync.  If we were collaborating using a Human Interaction Management System (HIMS), such as humanedj itself, none of this would be an issue.  The technology underpinning each communication would be irrelevant, since all would be conducted via a HIMS that automatically recorded events.

Then there is the nature of each conversation.  Most people want to talk about two separate things:

1. Their experience of using the software - its strengths, weaknesses, what they would like to see added, and what they would like to see removed.
2. What sort of relationship they can expect with my company Role Modellers if they adopt the software - issues of licensing, support, partnering, and integration.

In HIM terms, these are separate "Interactions" (goal-directed communication channels) in a single "Story" (collaborative work process).  Keeping them separate has all sorts of advantages.  In particular, one can then involve different people in each Interaction: techies for the software discussions, business people for the relationship discussions.

Finally, collaborations tend to evolve over time.  Just to give one example, several of the beta testers have now decided they would like to collaborate amongst each other.  There are several such sub-groups forming - some in particular sectors, some organized geographically, some for the purposes of producing written material such as articles, some to investigate particular ways of jointly testing the software itself.  A few of these collaborations are being conducted via humanedj, though not all.  It would certainly simplify matters if the creation, merger and splitting of Stories was managed using humanedj - since otherwise those involved have only an informal idea of what is going on.  Using humanedj to conduct the Stories would not only facilitate the work itself and allow it to be automatically recorded (as described above), but make it amenable to management control - others in the organizations concerned would have visibility of the work, and a means both of controlling and of supporting it.

TAKE AWAY

Do the examples above sound familiar to you?   A software trial program is a typical example of what HIM calls a "human-driven" process.  Perhaps some of the tasks may be automated, but the work is quintessentially about people and their interactions.  With such work, you cannot say at the start how things will turn out - indeed, a fundamental part of the work itself is to establish how things will turn out.

With the rise of what author Daniel Pink calls "Asia, Automation and Abundance", such work is becoming more important than ever, as the only true competitive differentiator left.  Further, the new Internet-based communication tools have left us all drowning in a sea of messages from others - what I call network overload - that can be neither handled by individuals nor controlled by organizations, let alone leveraged properly for advantage.

We all need to structure our communications better - as collaborations, in which the individual Interactions are understood, separated out, and supported by a new generation of software tools.  Ask your software vendors what they are doing about HIM - and if their answer is unhelpful, ask yourself whether you are working with the right software vendors.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications | Permalink | Comments (0)

In recent posts I have been considering what sort of software is required to properly and securely support human collaborative work - and giving examples of situations in which current tools and techniques fall down.  Here's one with which we are all familiar, yet which is in fact completely inappropriate in any real sense: the use of email to conduct a "business interaction": a conversation, dialogue, discussion, negotiation or any other set of inter-related work communications.

Email must be far and away the most common of all Internet business tools, more so even than the Web.  Not every company offers a transactional Web site, or uses such sites to do its own business.  But they all provide their employees with email and expect them to use it.  It is surprising, then, that when you actually start to think about it, that email turns out to be incredibly ill-suited to business use.

For a start, email is dreadfully insecure.  Some companies provide their employees with secure messaging services, but many more don't.  In general, emails are plain text communications that can be read by any intermediary as they travel across the Internet.  Further, you don't even know who is reading the email once it gets to its destination - it could be the person you address it to, or it could be their PA, or someone else who they have asked to check their email while they are out of the office.  It could also be an IT staff member with privileged access to the email system, or any superuser charged with maintaining it.

Email has in fact many technical security problems with falsified headers, corrupted attachments, and so on.  But let's put the insecurity of email aside, since it is such a huge and obvious problem - and look at some of its more subtle and interesting problems - ones that are business-related rather than technical.

First, there is tone.  In the early days of the Internet, its users recognized that email brought with it problems of emotional content.  Basically, when you write something you do so with an attitude - which may be anything from humorous to gently reprimanding to insistent to deferential.  The trouble is, the person reading it doesn't always interpret your message in the same way you meant it.  It is very easy to mistake the emotional content of an email, which is why early Internet users developed a concept of "netiquette" - a set of practices that enabled parties to a discussion to make their intent clear in terms both of logical and of emotional content.  These days, however, the use of email has spread and netiquette has vanished.  I've seen many an email discussion that would have been cordial if carried out in person deterioriate as the people concerned misread the tone of messages from each other.  And this can have serious business impact.

Second, there is involvement.  The people engaged in an email interaction tend to change as the conversation progresses.  Often people are CCed by one respondent but not by another - sometimes deliberately, other times just because a party presses "Reply" instead of "Reply All" by accident.  This gives rise to all sorts of thorny problems, not just of politics and wounded feelings, but very practical things such as people expecting someone to know (and act on) something that they were in fact never informed of.

Third, there is sequencing.  People tend to work through their inboxes in date order - oldest first.  So how many times have you replied to an email, promising certain actions, then realized there was a later email (perhaps not even from the same person) which renders your response inappropriate or even unnecessary?  To avoid this problem, I try to consciously force myself not to reply to anyone at all until I've read all my emails, which is a tiring and unnatural way to work.  Effectively, you end up reading everything twice, since once you've worked through your inbox once you have to start all over again - and hope that another email doesn't arrive while you're doing so!

Fourth, there is filing.  How do you file your emails?  Myself, I try to put them in folders, organized logically, but often the subject matter of the folders.overlaps.  Do you put a message from a consultancy client about a technical issue under "consultancy" or under "technical"?  Do you keep each client's email in a separate folder or organize it by "type" in some way?  Not only do such filing strategies rarely work as you would like, but they are also incredibly laborious.  It takes ages to set up and maintain the rules that assign messages to folders, and even then you end up doing some by hand as most email systems file incoming but not outgoing email.  One solution is just to leave everything in a default inbox and rely on search tools, but then what do you search on?  Every email has different keywords.  We're into the rarefied domain of what is called "latent semantic analysis" - and all we need is to store our interactions with people in a sensible way!

Fifth, there is the biggest problem of all.

Email is so easy to use that it's easy to overlook how for many people it is more of a problem than a convenience.  In a corporate environment, it is not unusual to receive hundreds of  emails per day, which both reduces productivity and increases stress. As they say, for some people email gets in the way of their work - and for the rest, email is their work.

Why do we all get so many emails, more than we can reasonably be expected to deal with? Because we never get the chance to specify exactly the messages we are willing to receive.

For example, many people routinely "CC the whole world" as a means of dealing with an issue that crops up - partly since it is easier than working out who exactly needs to know about the issue, and partly to cover their own backs. And the impact of this very common practice is enormous. Faced with an inbox full of such blanket postings, you are nevertheless forced to read each one carefully, since you can never be sure that the one message you skim, or skip entirely, is the one that raises a business problem for which you will later be held responsible. Far from increasing co-operation amongst colleagues, unrestrained messaging not only reduces productivity but also fosters a culture in which people are encouraged to offload issues onto others - rather than a culture in which people are motivated to personally ensure solutions

Similarly, when someone has a question that needs answering, the general assumption in a modern workplace is that one can just fire off an email to get the information required. No matter that the recipient(s) may already be drowning in work, or may not be the best people to ask anyway - simply by receiving an email, they each take on the responsibility to respond, even if it is only with a suggestion that someone else would be better placed to help out.

TAKE AWAY

The underlying problem with email is that people are rarely clear about who they are working with, and on what, or who is the best person to approach in a given situation. There is no clear visibility of shared and individual goals and responsibilities, so people deal with things in a variety of unpredictable ways.  For instance, they may spread the net as wide as possible, and include anyone that could possibly have a tangential interest - with the result that everyone's workload increases, general efficiency drops, and stress levels rise.

The only way forward is to provide a simple means for people to declare what exactly they are interested in, and what exactly they are responsible for - i.e., software tools that facilitate a more intelligent form of human collaboration.  Email does not provide this, and is never going to provide this.  It is a low-level protocol that should underpin more business-oriented tools for human collaboration.

And once we get such tools, perhaps we'll be able to get more work done, with less stress - and still get home in time to see the kids.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Office Applications • Security | Permalink | Comments (4)

In the next few blog entries I will be discussing open source projects that have reached a certain stage – that of being at least as mature as their commercial competitors. Many organizations are still reluctant to use open source software for mission-critical applications – and in some cases there are good reasons to be suspicious. I will conclude this series of blog entries with a discussion of the potential hazards of adopting certain types of open source software in an enterprise infrastructure.

However, these hazards are not those typically those voiced as objections by IT management – unstable or insecure software, availability of support, and legal issues. The open source projects I will be discussing in the next few entries are perfectly viable from all these perspectives. In general, major open source software applications are written at least as well as leading commercial products (often by the same people), enthusiastically supported by expert and helpful developers (as opposed to knowledge-free call center staff), and transparently licensed (via industry-standard agreements).

Similarly, the advantages of open source are not those typically quoted, either. For example, open source evangelists, being technical folk, make a big deal out of being able to correct or enhance the code yourself if you need to. This is exactly the opposite of what an enterprise is looking for - the very last thing they want is the headache of untangling some huge and complex application when there is a problem. The real benefit of open source is that it is generally more attuned to real-world needs, since open source projects get started precisely in order to meet such needs. While software vendors are grappling with legacy products and market positioning, the open source community just goes off and builds useful stuff.

Stay tuned to this blog for a discussion of the true hazards of open source as regards enterprise adoption, which are to do with the type of project. For now, I will be looking at projects that are not only free from such hazards, but that offer significant advantages over their commercial rivals.

First up is an offering in a space of increasing importance – information retrieval. The success of Google is a measure of how valuable this technique has become in modern life, yet despite the continuing advance of Web search, the facilities that most organizations offer for searching their intranet, or even their public-facing Web site, are little more than pitiful. The ability to retrieve a particular document from the sprawling Web presence (internal or external) of a large company is often more a matter of art than science, and may well depend on knowing in advance the likely path to the data concerned.

This is, of course, a well-known problem, to which knowledge management techniques are often touted as the answer. Indeed, such advanced solutions can be employed to unlock the information archives of a company – but you do not always need the sort of high-end, and very expensive, tools sold for this purpose. For one thing, there are alternative approaches to knowledge management that can be leveraged to expose the knowledge hidden inside information, some of which I discussed in previous blog entries, and I will be returning to this topic in future posts. However, there are also far simpler ways by which the hundreds of thousands of documents available via HTTP on a large company’s servers can be made available – and thus turned into an asset rather than a liability.

I was struck recently by the marketing puff for a commercial search engine, which offers as a “step forward” the ability not only to search structured data (requesting specific values for specific fields) but also to provide flexibility via assigning weights to the different terms in a search. Surprisingly, whoever wrote this PR spiel – and possibly the vendor itself – does not seem to be aware that such “advanced” features have been available for years in the leading open source search engine.

The search engine in question is Lucene, an Apache project. Lucene is not only very well-established but can do some very cool things. For example, Lucene can search via named fields - a feature that in itself offers a step on the way to full knowledge management – and offers wildcard, fuzzy, proximity and range searches. Terms in any of these types of search can be boosted, grouped, and controlled via the use of logical operators.

For a proper explanation of these features, see the links referenced above. The point is that Lucene is very full-featured. However, Lucene is not a search application – it is simply a search engine, a code library that can be used to interrogate any body of text. To use Lucene as the search tool on a Web site, for example, it must be embedded into a product designed for the purpose.

Fortunately, since June 2005 there has been a sub project of Lucene aimed at doing precisely this. The Web crawler and search facility that incorporates Lucene is Nutch. Nutch operates somewhat like Google - in fact Nutch incorporates some technology that Google itself put into the public domain, technology that permits Nutch to crawl, index and search enormously large collections of documents. Moreover, the user interface of the Nutch search facility is, if anything, more helpful than that provided by Google, providing an analysis of how the page ranking was generated along with the results themselves.

If I were a search engine vendor, I would be worried now that Nutch is getting off the ground properly. I have used Nutch in anger, and cannot see any reason to buy commercial software for this purpose now. Simple to install, robust, scales, configurable - what more could one want?

TAKE AWAY

If you work for a large organization, ask yourself whether its public Web site and intranet provide genuinely acceptable search facilities. If not, consider implementing Nutch. It takes – literally – minutes to do the initial installation, and configuration even for a large document base is not complex. And Nutch, like Lucene, comes from the Apache Foundation, one of the most (if not the most) well-established and reliable homes for open source projects.

Tune in next time for a discussion of other open source software with the potential to transform your enterprise IT environment for only a small investment of time and effort.

Posted by keithhb in Internet • Knowledge Management • Open Source | Permalink | Comments (2)

This is the concluding entry in a 5-part series on BPM futures. Previous blog entries showed that:

1. The imminent provision by the OMG of a standard XML format for BPMN will effectively render BPEL unnecessary
2. The adoption of BPMN for process definition means that business process automation will become a high-level graphical interface to component technologies, rather than a separate layer in enterprise architecture
3. By taking a computer science perspective, we can see that in due course Web service choreography will supersede Web service orchestration for the definition of those processes in which human involvement is limited to key decision and data entry points (what I call "mechanistic" processes)
4. Hence, forward-looking organizations need to consider now the creation of choreography descriptions, an activity for which the techniques of Human Interaction Management can be used: both to develop the processes needed to establish agreement on multi-party collaborative contracts, and to provide the graphical notation needed to depict the contracts themselves.

In this final blog entry (for now) on BPM futures, I will look at the type of software vendors best positioned to take business process automation forwards.

The argument above shows that business process automation is turning into a rather different sort of activity than has been supposed to date. As implemented by current BPM suites, business process automation is largely a matter of drawing flowcharts - effectively it is a sort of high-level, graphical programming. In fact, BPM competes directly with a combination of the UML and J2EE, for example. Though the picture painted by BPM vendors, and some analysts, is rather different, the battle could be considered a David vs Goliath one. I would happily bet that the total number of all BPM installations ever is nowhere even close to the average number of downloads of JBoss (say) in a single month - which in November 2005 was about 200K.

Of course, a BPM suite provides features that do not always come for free with a component-based approach to process automation - in particular, Business Activity Monitoring (BAM) - though here too technologies such as JMX are starting to provide a viable alternative. However, my point is simply that to date BPM has not become an ubiquitous part of enterprise technology, and this is because it is competing with other approaches that are mature and very well-established. Moreover, the people who are in the end responsible for implementing and maintaining BPM - the IT department and their outsourcing suppliers - are a lot more comfortable with J2EE or .NET than they are with a particular proprietary BPM toolset.

However, all this may change with the next generation of BPM tools. The argument that was made in previous blog entries, and that is outlined above, shows that business process automation will inevitably move towards the creation of multi-party collaborative contracts - where the parties may be internal to a single organization or span several organizations - and the use of these contracts to generate executable business processes. This is a very different kettle of fish from the usual software development process. Though agile methodologies, in particular, place an emphasis on negotiation and re-negotiation of client expectations, the future focus of BPM will be on the creation of agreements among several collaborating parties. BPM tools will then provide the means for each party to automatically turn their part in such agreements into working software.

Tools of this nature bear little relation to existing BPM products. But they do bear a strong relation to another class of products - those provided by vendors specializing in middleware to support Service-Orientated Architecture (SOA). SOA techniques are in flux at the moment, and there is a lot of debate among experts as to the benefits of a distributed vs centralized backbone, or the WS-* stack over simpler techniques such as REST and XML-over-HTTP. However, despite all this debate, there are some tenets to which most people adhere - in particular, the key technique of SOA is to establish what are generally called service contracts. There is little agreement among those in the field even on what a service is, but most agree that a service should expose some kind of description as to what it does. A service contract is a description of itself that a service publishes (somehow) - it is used by those that wish to call the service to determine what they can expect from it.

There is a strong parallel here to the direction in which business process automation is going. Both are based on the definition of contracts rather than on the definition of internal behaviour. SOA tools have not yet advanced to the point of being able to generate the latter from the former, but (as shown in the previous blog entry) there is every reason to suppose that this can and will be automated by software in due course.

There is a growing realization that the architectural principles emerging for the design of business processes by analysts (based on establishment of multi-party service contracts for collaborative behaviour) are the same as those already used by technicians in SOA. In fact, there is a strong argument that in a decentralized, globalized, Internet-enabled economy it will soon be hard to survive without applying such principles to the organization of your enterprise. As McKinsey wrote recently:

"the shift from transactional to tacit interactions requires companies to think differently about how to improve performance - and about their technology investments. Moreover, the rise of tacit occupations opens up the possibility that companies can again create capabilities and advantages that rivals can't easily duplicate."

Hence, it is realistic to expect that the next generation of tools for business process automation will come neither from platform vendors, nor from BPM pure-plays, but from vendors currently providing a complex, and to many people invisible, part of the IT middleware infrastructure - tools that come under such confusing and often overlapping categories as Web Services Management (WSM), Enterprise Service Bus (ESB), Enterprise System Management (ESM), and so on.

TAKE AWAY

In the next year or two, expect SOA to become high-profile among business leaders as well as technical folk - to become as much a management methodology as an approach to IT. This change will bring with it some key questions, related not only to development - how can we create a set of services, and agree on them with those responsible - but also to governance - what policies, procedures and other controls do we have in place for maintaining and regulating services? The SOA community is evolving some answers to these questions, but inevitably these answers are currently too IT-focused. If SOA is to successfully make the transition into a management approach, it needs to take on board the general principles and patterns that underpin business activity.

Such principles and patterns need not only to encompass diverse fields of business theory (quality management, knowledge management, project management, ...) but also insights from the many scientific disciplines connected with organizational life (psychology, biology, social systems theory, learning theory, and so on). Further, it is no good assembling such insights into a huge, diverse and confusing set of "best practices" - the consultant's favourite trick. They must be organized properly into a theory with a formal basis that provides the reassurance of sound thinking together with a structured and maintainable approach to implementation.

With respect to IT industry support for this move, expect SOA vendors to find an exciting new field opening up for them - the full support of mechanistic business processes, currently considered the domain of BPM tools. Those SOA vendors that embrace the move to declarative expression of processes descriptions, and automatic generation of process implementations, may well find themselves catapulted to a new and prominent level of visibility among business leaders.

PS

On a personal note, before publishing the ideas in this series I did not expect them to be met with much (or any) agreement. BPEL in particular is a focus of current IT industry interest, and consequent vendor investment. So it was a pleasant surprise not only to find that some influential people have already come to similar conclusions about BPEL - Phil Gilbert of Lombardi, for example, in his own blog - but also to receive many messages of support for the ideas in general. Whether you agree or disagree with the views expressed in the series, please let me know, either publicly via a blog comment or privately via email (see my bio). Private messages will of course be treated in confidence.

In the next blog entries, I will be looking at some of the most important open source software tools. I am continually surprised in my consulting work to find that organizations are paying for software that is less mature, less feature-rich and less well supported than that provided by the open source community. I will also discuss some of the situations in which open source is not a good idea.

Posted by keithhb in Business Process Management • Internet • Knowledge Management • Management • Service-Orientated Architecture | Permalink | Comments (1)

In the previous blog entry, I discussed the limitations of the RDF stack from W3C for knowledge management (KM) in business (if you're new to this blog, it would be worth reading that entry before continuing). This is not to say that the RDF approach is somehow “wrong” – far from it - just that it is worth exploring alternative approaches to KM. And, as we shall see, there is such an approach which not only has its own advantages but for which powerful open source tools are emerging.

The approach in question is Topic Maps. Topic Maps originate from OASIS, and have been ratified as an ISO standard, with an XML representation known as XTM (XML for Topic Maps). Roughly speaking, it is possible to represent most knowledge either using the RDF stack or using a Topic Map – techniques have even been proposed for automatic conversion from one format to the other, though there are technical difficulties inherent in doing this. However, the intentions of the two approaches are different, and it would be more sensible to view Topic Maps as complementary to the RDF stack, than as interchangeable with it.

In fact, the RDF and Topic Maps standards are of about the same vintage, dating in the first case from about 1999 and the second from about 2000. The two techniques were developed in parallel, by teams unaware of each other’s work, and each team looked at the problem of knowledge representation from quite a different angle.

RDF is essentially an implementation technique for the Semantic Web: “The RDF specifications provide a lightweight ontology system to support the exchange of knowledge on the Web”. Topic maps, on the other hand, are focused heavily on making it easier to find what you are looking for: “Topic Maps is a very explicit form of knowledge representation and it forces you to consider some of the issues that are critical when it comes to optimizing findability. It focuses a lot on how to name and how to identify subjects.”

In a nutshell, RDF is aimed at organizing information resources into predefined categories, whereas Topic Maps are aimed at exposing the internal structure of a particular body of information. If RDF is for directories, Topic Maps are for indexes.

Topic Maps are in some ways cuter than RDF. The principles are few and simple, yet enormously sophisticated structures can be created using them, some of which are hard to represent using the RDF stack without invoking very heavyweight techniques. Despite this, there is far less tool support available for Topic Maps. However, there are enough tools available – including open source ones - to make it worth considering Topic Maps for business use.

Why? I discussed in the previous blog entry how RDF-based tools have two particular disadvantages for business use – that they tend to categorize information into a hierarchy rather than a more flexible network structure, and that it is difficult to keep track of “old” knowledge (necessary to justify decisions or re-use knowledge, for example). Topic Maps have neither of these limitations.

The basic elements of Topic Maps are Topic (a subject or category), Association (a relationship between two or more specified topics), and Occurrence (information relevant to a specified topic). It turns out that this humble basis is enough to create complex webs of knowledge – mainly because the concept of Topic is general enough to represent any “thing” whatsoever. A topic is an information resource in its own right.

Hence, Topic Maps are not created via annotating each document in a corpus of information. Rather, a separate XTM file, embodying the network of relationships interwoven into the documents, is generated to represent the knowledge they contain. This means that it is simple both to categorize data into a network (as opposed to a tree), and to preserve previous states of enterprise knowledge (just keep the old XTM files you generate). Both are significant advantages in knowledge work, and are easier to implement with Topic Maps than with the RDF stack.

A nearly complete list of open source Topic Map software shows tools with various different applications, including graphical visualization, omitting only one other tool that I know of. A complete Content Management System based on open source Topic Map tools and an XML database is described by the IVS research group. A freeware (but not open source) mind mapping tool that imports and exports Topic Maps is ThinkGraph, and this is an area to watch – there are interesting analogies between concept mapping and Topic Maps, though mind mapping tools tend to view topics in a tree structure rather than as a network.

TAKE AWAY

The enterprise looking to develop a KM strategy and/or reduce its current expenditure on high-end commercial KM applications would be advised to consider Topic Maps as a representation technique, either as an alternative or a complementary approach to the RDF stack.

In future blog entries I will discuss open source tools that can be used to automatically generate Topic Maps from unstructured data – including the Web. Before doing so, however, I will be turning to the world of Business Process Management, looking at deep changes that are on the way, and showing how to prepare for them.

Posted by keithhb in Knowledge Management | Permalink | Comments (0)

Welcome to a new blog. You may already know of me from previous writings, either in print or on IT Web sites (which might be why you’re reading this) - but if not, I am a researcher, consultant, and software developer focused on new directions in Business Process Management (BPM). I am also CTO of Role Modellers, a company whose mission is to enhance understanding and computer support of collaborative human work processes in industry.

So this blog will cover such issues. However, it will also range rather wider, and attempt to drive out the underlying trends in enterprise technology, by:

• Taking a fresh look at conventionally accepted wisdom and predictions
• Placing emerging technologies into perspective
• Providing a hands-on guide to important alternatives to mainstream technology, including up-and-coming open source projects.

If you are responsible as CTO, CIO, architect, developer or consultant for helping determine IT strategy, this blog will help you make more informed decisions.

So let’s start with a current hot topic – enterprise knowledge management. McKinsey identify “learning how to leverage knowledge” as one of their top 10 trends for 2006 - and recently I have been working with UK government scientists to help improve the ways in which they collect and analyze information about current scientific research. This has led to some interesting discoveries - both about alternative techniques to those usually employed for knowledge management (KM), and about emerging open source tools.

Since KM can be a confusing field, let’s introduce the key concepts before going further. Visit the Web site of leading KM vendors such as Autonomy or Entrieva and you will find some very high-powered, feature-rich and hugely expensive product suites, claiming to help you “understand the hidden 80%” , “make sense of unstructured information”, “empower the knowledge worker”, and so on. What does all this actually mean in practice?

In general, KM tools are focused on the "annotation" of documents with "tags". In other words, hidden fields are added inside the documents that indicate:

• The sorts of thing being discussed (categories in which the information fits)
• Relevant connections (links to other information with similar categories).

Annotations can pertain to the entire document or just parts of it – particular word phrases or multimedia objects, for example. Depending on the toolset and the customer requirements, the fields can be created manually, semi-manually, or automatically. Once a corpus of documents has been annotated, it can be viewed in various new ways to try and better understand the nature of the information, for example via a graphical depiction of the inherent categories and links.

This sounds great, and some KM vendors have been very successful. However, leading experts tend to describe the success of KM technology as “dismal” . Why?

As the interview referenced above explains, management practices lie at the heart of the problem. New technologies on their own will not improve working practices. I will have a lot more to say in future blog entries about how radical new techniques for process management can make a difference here. For now, though, let’s stay with technology, and look at an alternative to the conventional approach to KM that may well offer more power in typical business situations – and for which open source tooling is emerging.

Most KM tools make use of the RDF stack – a set of standards from the W3C that support XML annotation of documents. Each of these standards (RDF, RDF Schema and the 3 sub-languages of OWL) is based on the preceding one, and provides more expressive power at the expense of added complexity. The details are all very technical, but there are two features of the RDF stack of particular business importance.

Firstly, the lower layers of the RDF stack (the layers used by most tools) support only hierarchical categories of information: taxonomies rather than networks. This means that such tools impose an unnatural ordering upon business knowledge. For example, Autonomy claims that their: “Taxonomy Generation feature can automatically and consistently understand and create deep hierarchical contextual taxonomies of information based on conceptual understanding” and Entrieva claim that their SemioSkyline product enables “users to browse or search for information organized by category within a familiar, hierarchical structure” [my italics]. However, most business knowledge is more like a network than a tree. Knowledge categories do not generally divide neatly into parents, children, grandchildren and so on – rather, links between them may occur all over the place. Even if multiple taxonomies are used together, business people do not really see their world in hierarchical terms.

Secondly, RDF annotations are often added into the documents themselves. For knowledge associated with HTML documents the RDF Model and Syntax Specification from W3C says, "The recommended technique for embedding RDF expressions in an HTML document is simply to insert the RDF in-line" (another common approach is to use "Dublin Core" metadata within the META tags in the HEAD element of the document). If such techniques are used, as you update the documents the annotations also get updated, and it is hard to track changes to your knowledge independently of changes to your documents. Why should you need to? Well, the fundamental principle of KM is that business decisions should be based on business knowledge. Therefore, if you want to understand why a certain decision was taken - to justify it, to re-use the reasoning, or for any other reason – you need access to the knowledge that was available at the time. The knowledge available right now is no use for this purpose - but with most KM tools, that is all you get.

So, is there another approach to KM, one in some ways more in tune with real business needs? And one for which open source tools are emerging? Tune in again to this blog for more about an alternative approach to KM …

Posted by keithhb in Knowledge Management | Permalink | Comments (1)