First session of the last morning, Eduardo Chiocconi of BEA and Rob Wald of JPMorgan Chase talked about the ALBPM UI: what comes out of the box, and what you can build yourself.

Out of the box, ALBPM has three user interfaces alternatives:

• HiPer WorkSpace, a full user workspace with menus based on their permissions, a view of their inbox, and views of any instances that the user opens. It uses CSS if you want to change styles, and can be customized further in terms of removing buttons and other controls.
• JSR-168 portlets for any standard portal environment, such as WebLogic.
• WorkSpace Extensions WorkList Portlets that can be plugged into ALI and provide additional integration functionality over the standard portal interface, such as integration with the ALI Collaboration environment.

They're working on consolidating these interfaces in order to reduce the user learning curve.

If those don't work for you, then you can create your own user interface, either browser-based or rich client, using the available APIs. For building web clients, many of their customers have used JSP to re-code the entire user interface, then used servlets to access the engine. Alternatively, you can use Struts or JSF/AJAX. All of these can use their Java-based process API, PAPI, or the web services version, PAPI-WS, to retrieve instances from the engine, or WAPI (a servlet-based API) to execute interactive activities such as screen flows.

For rich clients, they're seeing a lot of .Net development that uses PAPI-WS to retrieve instances, then create the UI. It's also possible to use Eclipse or Swing to build rich user interfaces that call PAPI directly. This is more complex for interactive activities, but there are ways to work around that.

To sum up, there are three public APIs:

• PAPI (process API), which is a Java implementation. If you're working in a Java environment, this provides the tightest integration and best functionality. It manages multiple process engines transparently, and does instance caching on the client side to reduce latency in connecting to the engine -- a critical performance factor.
• PAPI-WS is a subset of PAPI that operates as web services (SOAP), although this is being extended in the near future to provide the full functionality of PAPI. There may be a .Net version of PAPI in the future, but for now you have to use PAPI-WS if you're developing in .Net (e.g., ASP.Net), and can also be used from any web service client. Right now, PAPI-WS is part of the HiPer WorkSpace, but will be decoupled as a self-contained web application in the future. It's also possible to expose processes directly as web services, as we heard in an earlier session, which provides another point of integration from a web service or .Net development environment.
• WAPI, which is a servlet API that can be used to launch the UI of an interactive activity at a point in the process, which can't be done in PAPI or PAPI-WS.

With any custom UI, there's always the question of single sign-on. With the WorkSpace Extensions WorkList Portlets in ALI, that's handled natively; in the HiPer WorkSpace and JSR-168 portlet implementations it requires some customization, although there is a single sign-on login servlet provided with the JSR-168 portlets to make this easier.

Getting to the specific JPMorgan case study, they created a custom user interface since, like many large companies, they want integration with other applications in their environment and want more control over the look and feel of the interface. It's possible to just create custom JSPs and use them in the standard work portal framework, which provides a great deal of control over the UI without completely rewriting it, but this wasn't sufficient for many of their applications. What they ended up doing was creating a completely custom inbox using Struts/JSP/GWT with PAPI: one example that he showed was using Struts and AJAX via the Google Web Toolkit to manage financial reconciliation processes. They're also using IceFaces, an open source RenderKit implementation of JSF (as a replacement for Struts) that supports AJAX to create a visual drag-and-drop components for use in an IDE such as Eclipse. Since JPMorgan is dedicated to the use of open source, they're doing some innovative development that's not seen in most corporate environments, but maybe should be. They're also using the JSR-168 portlets in a more standard portal implementation, and building rich clients with Eclipse.

On the back end of their implementation, they've found that some of the PAPI protocols don't work well over wide-area networks, such as between their US and Japan operations, so they do quite a bit of preloading of the PAPI cache.

JPMorgan has implemented ALBPM as a centralized shared service in order to provide efficient use of both human and server resources: centralized code and best practices on the human side, and a single ALBPM server handling 10 applications without difficulty.

Posted by Sandy Kemsley at 10:56 AM in BEAparticipate • BPM • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

I'm jumping around between tracks (and hence rooms): I started the afternoon in the ALUI Experience track, then on to the ALBPM Technical/Developer track, and now I'm in the ALBPM Experience track for a discussion of best practices for managing BPM projects with Dan Atwood of BEA (another former Fuego employee) and Karl Djernal of Citigroup. It's a bit difficult to pick and choose sessions when you're interested in multiple tracks: this session and the one after it in the same track are both around best practices, although appear to cover different aspects of BPM implementations and I'd like to sit through both. This one is categorized as "beginner" and the next as "intermediate", so I'm hoping that someone's actually checked to ensure that there's not too much overlap between them. I'd also like to see the next technical track on how BPM and ESB work together, but think that I can probably get a briefing on that directly from BEA as required.

Atwood started the session with seven key practices for BPM success:

1. Fundamentals of process-based competition: understanding the competitive advantage of being a process-oriented company, and the business case for BPM.
2. BPM and its value to the corporation: understanding what BPM is and how it differs from other management and technology approaches.
3. From functional management to process-oriented thinking: how the shift from functional management must permeate through the ranks of middle management in order to disperse the fiefdoms within an organization.
4. Getting hands-on BPM experience, with the help of mentors.
5. Foundations for process practitioners: BPM as the capability for implementing and extending other management practices such as Six Sigma.
6. Business process modelling and methods: learn about process-oriented architectures and development methods, and how they differ from traditional approaches.
7. Human interactions and their roles within BPM: while system-to-system automation is often a BPM focus, the human-facing parts of the process are critical. In other words, you can't think of these as being "human-interrupted" processes, as a customer of mine did long ago.

Obviously a big fan of BPM books, Atwood references Peter Fingar, Howard Smith, Andrew Spanyi, John Jeston, Mike Jacka, Paulette Kellerin and Keith Harrison-Broninski, as well as a raft of BPM-related sites (although not, unfortunately, www.column2.com). Also a fan of lists, he finishes up with his top five success factors:

• Executive sponsorship
• Correct scoping
• Start with the end in mind
• Framework
• Engage stakeholders

Hmmm, that seems to make 12 best practices in total...

Djernal then discussed the Agile methodology that they used for BPM implementation at Citigroup, starting with a description of Agile and Scrum as the anti-waterfall approach: providing incremental deliveries based on changing, just-in-time requirements, and involving the end users closely during the development cycle to provide feedback on each iteration. Just as important as delivery mechanisms is the Agile team structure: the team's not managed in the traditional sense, but works closely with the customer/end-user to create what they want. There's a 15-minute team meeting every day, and a delivery (sprint) every 30 days. Many teams vary the sprint length slightly while sticking to the Agile methodology, although there's danger in increasing it too much or you slip back to months-long delivery cycles. Initiated by the original prioritized set of product features, the user feedback on each iteration can impact both the features and the priorities. There's basically three roles in Agile: a product owner who represents the stakeholders, the team that implement everything, and the ScrumMaster who provides mentoring on the Agile process and helps to sort out external roadblocks.

The interesting thing is how they brought together BPM and Agile, since I'm convinced that these are two things that belong together. Process diagrams fill in a lot of the documentation gap and are a naturally agile form of creating a functional specification; they form a good basis for communication between the business and IT. Changes in requirements that cause changes to the business process can be done easily in a graphical process modelling environment. In fact, in many BPM environments, the processes can be prototyped and an initial executable version developed in a matter of days without writing any code, which in turn helps to set priorities on the functions that do require coding, such as developing web services wrappers around legacy systems.

They've learned some things from their experiences so far:

• Get training on using the BPM products, and on BPM in general.
• Use some external resources (like me) to help you get started.
• Since BPM involves integration, setting up the development, testing and production environments can be time-consuming and require specialized resources.
• Spend some time up front putting together a good test environment, including automated testing tools.
• Create a centre of excellence for BPM.
• Start something small for your first BPM project.

There's a lot of arguments about how Agile can't really handle large-scale development projects, but it's my belief that most BPM projects lend themselves well to Agile. The worst disasters that I've seen in BPM implementation have been the product of the most non-Agile development processes imaginable, with months of requirements writing followed by many more months of development, all of which resulted in something that didn't match the users' requirements and was much too costly to change. As I've said many times before, if you can't get something up and running in BPM in a matter of a couple of months, then you're doing something really wrong.

Posted by Sandy Kemsley at 05:47 PM in BEAparticipate • BPM • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

David Messerschmitt, a prof at UC Berkeley and the Helsinki University of Technology, finished the formal presentations for the day with a talk on how inter-firm cooperation can be improved in the software industry. This is an interesting wrap-up, since we've been hearing about technology, applications and business opportunities all day, and this takes a look at how all these new software industry companies can cooperate to the benefit of all parties.

He started out by proposing a mission statement: focus the software industry's attention and resources on providing greater value to the user and consumer. This has two aspects: do less harm, and do more direct provision of value to the customer rather than the computational equivalent of administrivia.

In general the software industry has a fairly low customer satisfaction rate of around 75%, whereas specific software sectors such as internet travel and brokerage rank significantly higher. In general, services provided by people have a lower satisfaction rate (likely due to the variability of service levels), and the satisfaction rates are decreasing each year. Complaints are focussed on gratuitous change (change due to platform changes rather than anything that enhances user value) and security, and to some extent on having to change business processes to match an application's process rather than having the system adapt to their business process. Certainly, there are lessons here for BPM implementations.

Messerschmitt raised the issue of declining enrolment of women in computer science, which he thinks is in part due to the perception that computer science is more about heads-down programming rather than about dealing with users' requirements. He sees this as a bit of a canary in a coal mine, indicating some sort of upcoming problem for the computing industry in general if it is driving away those who want to deal with the user-facing side of software development. Related to that, he recommends the book Democratizing Innovation by Eric von Hippel, for its study of how customers are providing innovation that feeds back into product design and development, not just in software but in many areas of products.

He ended up by discussing various ways to improve inter-firm cooperation, such as the Global Environment for Networking Innovations (GENI) initiative, ways to accomplish seamless operation of enterprise systems, and referring to a paper that he recently wrote and will be published in July's IEEE Proceedings, Rethinking Components: From Hardware and Software to Systems. He then listed elements of collective action that can be pursued by industry players, academia and professional organizations to help achieve this end:

• Systematically look at knowledge gaps and ensure that the research is addressing those gaps
• Create/educate the human resources that are needed by the industry
• Understand and encourage complementarities, like broadband and certain types of software
• Structures and processes: capture end-user innovations for incorporation into a product, and achieve a more orderly evolution of technology with the goal of leaving behind many fewer legacies in the future

He's definitely of the "a rising tide lifts all boats" mindset.

Posted by Sandy Kemsley at 08:32 PM in NewSoftwareIndustry • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Bob Glushko, a prof at UC Berkeley, and Shelley Evenson, a prof at CMU, discussed different views on bridging the front stage and back stage in service system design. As a side note, I have to say that it's fun to be back (temporarily) in an academic environment: many of these presentations are much more like grad school lectures than standard conference presentations. And like university lectures, they cover way too much material in a very short time by speaking at light speed and flipping slides so fast that there's no time to even read what's on the slide, much less absorb or document it. If I had a nickel for every time that a presenter today said "I don't have time to go into this but it's an important concept" while flipping past an interesting-looking slide, I could probably buy myself the drink that I need to calm myself after the information overload. :)

Glushko posits that greater predictability produces a better experience, even if the average level of service is lower, using the example of a self-service hotel check-in versus the variability of dealing with a reception clerk. Although he doesn't mention it, this is exactly the point of Six Sigma: reducing variability, not necessarily improving service quality.

He goes on to discuss the front stage of services, which is the interaction of the customer or other services with the services, and the back stage, which is the execution of the underlying services themselves. I love his examples: he uses an analogy of a restaurant, with the front stage being the dining room, and the back stage being the kitchen. Front stage designers focus on usability and other user interface factors, whereas the back stage designers focus on efficiency, standardization, data models and the like. This tends to create a tension between the two design perspectives, and begs the question if these are intrinsic or avoidable.

From a design standpoint, he feels that it's essential to create information flow and process models that span both the back and front stages. The focus of back stage design is to design modular and configurable services that enable flexibility and customization in the front stage, and to determine which back stage services you will perform and which you will outsource/reuse from other service providers. Front stage design, on the other hand, is focussed on designing the level of service intensity (the intensity of information exchange between the customer and the service, whether the service is human or automated), and to implement model-based user interfaces and use these models to generate/configure/specify the APIs of user interfaces for the services. By exposing back stage information in front stage design, more back stage information can improve the immediate experience for a specific customer, and can improve subsequent experiences. Data mining and business intelligence can also improve service for future customers.

Evenson, who specializes in interaction design, has a very different perspective than Glushko, who focusses on the back stage design, but rather than being opposing views, they're just different perspectives on the same issues of designing service systems.

She started out with a hilarious re-rendering of Glushko's restaurant example by making the point that she applied colour to make the division of the co-production between front and back stage more visible.

Her slides really went by so fast that I was only able to capture a few snippets: sensors will improve the degree of interaction and usefulness of web-based services; technology influences our sense of self; services are activities or events that form a product through interaction with a customer; services are performances: choreographed interactions manufactured at the point of delivery; services are the visible front end of a process that co-produces value. A service system is a framework that connects service touchpoints so that they can sense, respond and reinforce one another. The system must be dynamic enough to be able to efficiently reflect the expectations people bring to the experience at any given moment. Service systems enable people to have experiences and achieve goals.

She discussed the difficulties of designing a service system, such as the difficulty of prototyping and the difficulty of representing the experience, and pointed out that it requires combining aspects of business, technology and experience. She feels that it's helpful to create an integrated service design language: systems of elements with meanings (that designers use to communicate and users "read") plus sets of organizing principles.

Posted by Sandy Kemsley at 07:20 PM in NewSoftwareIndustry • SOA • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Martin Griss of CMU West and Adam Blum of Mobio Networks had a fairly interactive discussion about integrating traditional software engineering practices into modern service oriented development.

Griss is a big proponent of agile development, and believes that the traditional software development process is too ponderous; Blum admits to benefits from smaller teams and lightweight process for faster delivery, but he believes that some of the artifacts of traditional development methods provide value to the process.

Griss' problems with traditional development are:

• Too many large documents
• It's too hard to keep the documents in synch with each other and the development
• People spend too much time in document reviews
• Use cases are too complex
• Can't react well to changes in requirements
• Schedule and features become omnipotent, rather than actual user requirements

In response, Blum had his list of problems with agile development:

• Some things really do need upfront analysis/architecture to create requirements and specification, particularly the lower layers in the stack
• Team management needs to be more complex on larger projects
• Many agile artifacts are simply "old wine in new bottles", and it' simply a matter of determining the right level of detail
• If you have a team that's currently delivering well, the introduction of agile processes can disrupt the team and impact productivity -- if it's not broke, don't fix it
• Some of the time-boxing of agile development (e.g., SCRUM monthly sprints, daily 10-minute meetings) creates artificial schedule constraints
• Agile development theory is mostly pseudo-science without many facts to back it up
• Modern tools can make older artifacts lighter-weight and more usable

Writing requirements and specifications is something that I've spent probably 1000's of hours doing over the years, and many of my customers still require this methodology, so I'm sympathetic to Blum's viewpoint: sometimes it's not appropriate or not possible to go agile.

An interesting point emerged from the back-and-forth discussion: it may not be possible to build the development platforms and frameworks themselves (such as what Mobio builds) in an agile fashion, but the applications built on those high-level platforms lend themselves well to agile development. Features to be added to the platform are effectively prototyped in an agile way in applications built on the platform, then are handed off to the more traditional, structured development cycle of the platform itself.

Griss, who was partially looking to just stir up discussion earlier, pointed out that it's necessary to take the best parts of both ends of the software development methodology spectrum. At the end, it appears that they agree that there are methodologies and artifacts that are important, it's just a matter of the degree of ceremony to use on any given part of the software development process.

Posted by Sandy Kemsley at 06:14 PM in NewSoftwareIndustry • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Michael Cusumano is with the MIT Sloan School of Management, and has written several books on the changing software industry; he spoke today about the changing business of software.

In general, there is a decline of new enterprise software product revenues, and growth in services and maintenance sales. There are a number of new business models, including SaaS and ad-supported software.

Software companies tend to move from being product companies to services or hybrid product/services companies (maintenance revenue is usually included in services). However, there's a different evolution curve that shows where companies focus on product innovation, then on process innovation (e.g., making the product more efficiently), then on services innovation.

The number of publicly-owned software companies peaked in 1997 at around 400 companies. IT services firms peaked in 1999 at around 500 companies. Web companies, which can be launched with significantly less capital (due to distribution mechanisms and development tools/methodologies), had a peak in 1999 before dropping in the crash, but are now climbing to an even higher peak.

Cusumano showed a graph of three business model dimensions: revenue model, delivery model, and customers, with traditional software product vendors at the origin of the graph, and various other models scattered throughout the cube. He also asked the question, is the rise in services and new business models temporary or permanent? The "temporary" argument says that we're in a transition phase between platform and business model innovations; the "permanent" argument (with which I agree) says that software is now commoditized and prices will fall to close to zero as we embrace SaaS and ad-supported models.

Being an MIT geek, Cusumano had slide after slide of data analysis about his research on software product companies. For example, average product company revenue crossed over in 2002 so that services revenue was larger than product revenue; also, firms at 24+ years of age have more services than product revenue. The age phenomena contributes to the date-based phenomena, since many of the large enterprise product vendors are reaching this level of age maturity now. There's an interesting cycle where services are very attractive for revenue generation, but then reach a point (in terms of % of revenue) where they are performed relatively inefficiently and, due to lower profit margins, are not as profitable as product; eventually, as companies become better at providing services (e.g., reusability), it swings to a more positive contributor to profitability. Market cap follows a similar pattern, although the centre (when services are undesirable) portion of the graph is broader.

Similar things are happening with hardware companies: more than 50% of IBM's revenue, for example, is from services.

He had some interesting comments on the way that software product companies should incorporate services into their business model: it should be planned and exploited as opposed to just happening by accident, as it does with many product companies.

He ended up with some key questions:

• How to manage the mix of products, services and maintenance efforts and revenue within a product company.
• How to "servitize" products, to make them less generic and more customizable.
• How to productize services; a great point that he made here is that it's best served by creating two professional services organizations with different mandates.

Posted by Sandy Kemsley at 05:48 PM in Enterprise2.0 • NewSoftwareIndustry • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Often, when I receive a request for a meeting on something that's far outside of my usual BPM/Enterprise 2.0 interests, I'll turn it down. However, when the meeting is with various deans and professors at Carnegie Mellon University West about their new Masters in Software Management program (press release here), I'm happy to make an exception. I graduated as an engineer over 20 years ago, and programs like this just weren't available then; I was curious to see how engineering education has advanced. I had a call with Dr. Jim Morris, dean of the CMU west coast campus, Dr. Martin Griss, associate dean for education and director of the software engineering program, and Tony Wasserman, executive director of the Center for Open Source Investigation. Of course, they're all professors at CMU as well, at the relatively new campus in Silicon Valley.

The Masters in Software Management is like a software engineering equivalent to an executive MBA: it's intended for people who are already experienced practitioners but want to improve their management skills in a big way, and do so part-time while they continue to work so that they can start to see immediate application and benefit. It grew out of the high level of interest in the management courses offered as part of the Masters in Software Engineering program that's been running since 2002, as well as interest from employers in the marketplace for the skills that they plan to teach. The Masters in Software Management is less technical than the Masters in Software Engineering, but offers some amazing courses that I think should work their way into any senior software engineering or computer science curriculum: open source, enterprise architecture, managing distributed teams, outsourcing, and many others. Since these are presented in a current business context, using long-running teams and simulating a small company experience, the goal is to produce the next generation of software leaders.

The program doesn't kick off until later this year, so they don't know the demographics of the student population yet, but are expecting that most will have a technical computer science/software engineering background, and that there will be a mix of those from small companies who want to improve their skills and build the next Google, and some from large companies who are either closet entrepreneurs or are serious about software management within their organization. About 1/3 of the Masters in Software Engineering program attendees are women, and they expect the percentage to be higher in the Software Management program. As in the Software Engineering program (where about 30% of the students are offsite), they'll allow remote students, although they need to be onsite for the 4-day kickoff and a few more times during the 2-year program.

Posted by Sandy Kemsley at 08:50 AM in OffTopic • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

I had an email yesterday from my friend Robb, which I have his permission to publish here. Robb used to work for me -- in fact, I think that I hired him a total of three times -- and whenever a company seeking to hire him calls me for a reference, I always tell them that the only negative thing about hiring him is that when I'm ready to start another company, I'll be hiring Robb away from them. Robb has four essential qualities when it comes to working: he's smart, he'll do anything to get the job done for the customer, he always has my back, and he's funny. His email yesterday, as usual, showed off the smart and funny bits:

Below is a newly formed company called Miria Systems. I give a history how this company came into being. Imagine how some things never actually die off:

• 1998 - A company named Application Partners was founded around an insurance/finance product and named it Sequis-- because apparently the English language is short of meaningful words
• 2000 - FileNet bought Application Partners and renamed it PeBA (standing for Panagon eBusiness Application) shortly after that they got bored with the name PeBA and renamed the product Acenza
• 2003 - After some successes the Acenza FileNet stalled the product program (to make way for BrightSpire) and effectively sold Acenza to a company called Endymion
• 2004 - Endymion changed the name from Acenza to (drumroll) Acenza for Payables
• 2003/2004 - Not satisfied with a simple named change Endymion completely re-wrote Acenza for Payables and changed the name to ManagedPay
• 2004 - Endymion apparently used up a lot of their budget on the name changes (and even more on re-write) because they were forced into a merger with Software Consulting Group (SCG) -- the two companies formed Soluziona USA
• 2005 - Not satisfied with a single re-write was good enough to keep their engineers happy Soluziona USA again decided to re-write ManagedPay but decided that the name should stay the same
• 2006 - Soluziona USA sold their ManagedPay product to a group of investors who formed a new company around this three-times-rewritten-and-five-times renamed product, keeping with tradition they gave the company a name with equally little English meaning as anything else in this brief history -- Miria Systems, below is the link

www.miriasystems.com

The interesting thing here to me here, besides the obvious snide comments around product/company naming exercises, is that the functionality of this product lives on despite name changes, rewrites, etc etc. That the market still has the (relatively) same problems almost ten years later makes me think that there is a disruptive or revolutionary solution waiting to happen.

No doubt you will recognize all of those companies and product names, most of which (except for Miria) I have been either directly or indirectly involved with.

Just some thoughts....

/rr

What Robb didn't mention is that PricewaterhouseCoopers also took the Acenza code base and rewrote it in J2EE around the time that they were purchased by IBM, to create a case-based application framework targeted at insurance customers. Once part of IBM Global Services, it was further rewritten to make it "vendor independent", meaning that the underlying content and process management could be either IBM or FileNet products, hence serving up the least common denominator of functionality and completely obviating proper use of the underlying product. I had the unhappy job of doing a review of an installation of this on behalf of the customer, and it's unbelievable how little of the FileNet product capability was actually exposed or used.

Although I agree with Robb that the market need for systems like this remains, I'm not sure at all that Sequis/PeBA/Acenza/ManagedPay/Miria product remains a viable solution. I haven't seen it in a while, but last time that I did look at it, it was too heavy and rigid, too old-school, and had the same problem as the IBM version that I mentioned above in that it completely hid the underlying capability of the BPM system below it: you might as well be using records in a database table for queue entries rather than a BPM system like FileNet.

I've talked about this problem of hiding the very nature and capabilities of a BPM product behind a rigidly-structured custom system in the past, and discussed it briefly on my Web 2.0 and BPM podcast, and I feel that it's a significant contributor to the lack of acceptance of BPM in many organizations.

I believe that the new world of enterprise software is less customization and more customizability: give the users the raw product and let them do what they need with it.

Posted by Sandy Kemsley at 12:52 PM in BPM • Business • FinancialServices • SoftwareDesign | Permalink | Comments (4) | TrackBacks (1) | Add to del.icio.us

I was on the treadmill this morning with Malcolm Gladwell. Actually, he was on my iPod, and I was watching his talk from TED 2004, which was posted recently on the TEDTalks site (not sure if the timing is correct -- although the website claims that this was recorded in February 2004, Gladwell mentions his book Blink, which wasn't published until January 2005).

The topic of his talk was spaghetti sauce, and although the words "long tail" are never mentioned, the story that he tells is about, to some degree, the lengthening of the spaghetti sauce tail: how a single style of tomato sauce in the 70's became the myriad styles that you find on your supermarket shelf today. This came about not by asking people what style of sauce that they wanted, since many had only ever been exposed to one type of sauce, but by offering them a huge variety of sauce styles and plotting the clusters of preferences, which turned out to be plain, spicy, and extra chunky. This in turn led to the explosion of styles of vinegars, mustards, coffees and many other food products, as the food industry learned a couple of valuable lessons:

Lesson #1: The way to find out what people want is not to ask them, since we can't always explain what we want, especially if we are completely unaware of what alternatives are possible.

Lesson #2: Different styles of products are not better or worse, just different. This democratized what might previously have been considered a hierarchy of product styles.

Although Gladwell's discussion about sauces convinced me that he has never had a "culturally authentic" Italian pasta sauce (my favourite is a pureed tomato and red pepper sauce that I learned to make at cooking school in Tuscany), he makes an excellent point about how the food industry was trying to treat us all the same, when we really wanted variability. As he summed up, "in embracing the diversity of human beings, we will find a sure way to true happiness."

All that I could think of as I listened was that the lessons learned by the food industry could be well applied in software design (you knew that I'd be getting around to something relevant soon). One of the major causes of bad system implementations is to allow the users to design the system based on their current knowledge, through a misguided notion of what a JAD session is supposed to achieve. I've had this experience many times over when introducing new technology such as BPM into an enterprise: the business users have no idea what such technology can do, since they've never used it or probably even seen it before, so it's foolish to expect that they are going to be able to tell you what they want the new system to do. Instead, they need to be presented with a vision of the future, or rather, several alternative visions, and be allowed to get their heads around what's possible in the new world that includes that technology. I'm not suggesting that the technology should reshape the business process to fit its limitations, but that the business processes can radically change -- for the better -- with the advent of the technology, if they are allowed to do so.

The lesson about variability is a good one to take to heart as well. Many implementation "experts" have a set of templates that they seek to apply to every implementation in which they are involved; this maximizes their profitability since they don't need to do much original design work for each project, but ultimately leads to unhappy users since it's not so easy -- or smart -- to make one organization's processes identical to another's. The diversity of processes both within and across organizations is part of what makes an organization unique, and their ability to create that diversity easily while maintaining a common business goal is what makes for an agile company.

Posted by Sandy Kemsley at 09:59 AM in BPM • SoftwareDesign | Permalink | Comments (1) | TrackBacks (1) | Add to del.icio.us

I've been a member of IEEE for over 20 years, and browse the periodicals that arrive at my door monthly, but have just become aware of the content that they offer to members and non-members on Spectrum Online. I've downloaded a number of the podcasts from Spectrum Radio and thoroughly enjoyed the two-parter about the enormously expensive and unsuccessful FBI case file project (I'd love to link to it directly, but they have a stupid Flash interface that doesn't allow that, so you'll just have to find it on the Radio page).

IEEE actually had to litigate to have the report about this project released under freedom of information laws, and the software experts in discussion on the podcast dissect the report and talk about what went wrong as well as lessons that can be learned for any large software project. Interesting that a company that has CMM level 5 certification can end up developing a $170M project with no proper requirements, no enterprise architecture, and a number of other things that seem like no-brainers for any large software project. Worth listening.

Posted by Sandy Kemsley at 11:27 AM in EnterpriseArchitecture • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

I've been watching the TED videos recently, and there's some very interesting stuff (who knew Al Gore could be that funny?), but this one of Jeff Han displaying a new type of user interface is amazing. If you're interested in data visualization, check out the Hans Rosling one as well.

Posted by Sandy Kemsley at 08:24 AM in SoftwareDesign | Permalink | Comments (2) | TrackBacks (0) | Add to del.icio.us

Veneer was great, but short, so I'm on to Sarah Harmer's You Were Here. I'm starting to slow down -- it's was a long day of travel yesterday and a long day of idea generation today -- so may not finish up all my posts today.

The next session for me was AJAX design patterns, which was good although focussed a bit too much on security issues for what I was looking for. Some great stuff on UI and performance issues. The wiki page has all the technical details, includes references to further JSON reading, so I'll just touch on some of the things that stuck in my mind during the session about AJAX UI design:

• Action of the back button: was the last user activity a navigation or an action? Can it be "undone" by navigating back, or is it appropriate to return to a higher level/home page?
• Action of the refresh button.
• URL and permalinks: appended # arguments that don't hit the server but are pure client processes to make the AJAX calls. Implications for search engines (agents can't index pages directly and would require an alternative representation to be referenced by robots.txt, but doesn't handle issues of relevancy through links), emailing permalink references.
• Tradeoffs between user experience and technical issues.
• Some actions need to be synchronous (e.g., "buy it" and other transactions), requires forcing synchronicity in AJAX or breaking out of AJAX for that part of the transaction.

Posted by Sandy Kemsley at 11:32 PM in SoftwareDesign • mashupcamp | Permalink | Comments (1) | TrackBacks (0) | Add to del.icio.us

This is brilliant: an article by Martin Fowler in IEEE Software magazine from a few years back (via Julian On Software) really nails the issue of agility and complexity by referencing, oddly enough, a speech given by economist Enrico Zaninotto at the XP 2002 conference. Fowler says:

One aspect I found particularly interesting was his [Zaninotto's] comment that irreversibility was one of the prime drivers of complexity. He saw agile methods, in manufacturing and software development, as a shift that seeks to contain complexity by reducing irreversibility -- as opposed to tackling other complexity drivers. I think that one of an architect’s most important tasks is to remove architecture by finding ways to eliminate irreversibility in software designs.

Most of my customers are large financial and insurance organizations that still use very waterfall methods for development. The requirements and functional design take months to develop, and have concrete poured firmly over them as soon as they are complete. In other words, the irreversibility starts at the requirements stage, long before development even starts. Of course, since a technical design follows the requirements stage and in turn is solidified before development begins, the irreversibilty is built into this stage as well: any changes have to go back through (potentially) several layers of approval and redesign, which impacts project schedules and contracts.

Fowler referenced an example where a database administrator made it easy to change the database schema and migrate the data for a project; as Fowler put it, "the database schema is no longer architectural" since it could be changed on the fly to accommodate the requirements of the project, rather than being a pre-supposed part of the design.

When we used to do this, it was called "coding by the seat of our pants"; now it's Agile!

Posted by Sandy Kemsley at 11:18 AM in SoftwareDesign | Permalink | Comments (2) | TrackBacks (0) | Add to del.icio.us

Phil Wainewright posted one of the IT commandments today: Thou shalt document all thy works. This is a perfect followup to my post yesterday about SOA and data, although it may not appear obvious at first. Problem: application developers don't use services when they should; in yesterday's post, I was talking about how they squirrel away data in their own application-specific silos, but the real issue is more widespread than that. Wainewright hits it on the head:

Failure to document is thus one of the biggies of ZDNet's IT Commandments, high up in the mortal sin rankings with the likes of 'Thou shalt not kill'. For if you don't document your work, how is anyone else supposed to reuse any of it? From your greater sin flows a multitude of others' lesser transgressions.

In addition to yesterday's more easily defeated arguments that developers don't use services because the data may not be accurate or it make take too long, we add this one that's harder to counter: the developers don't use services because they're not properly documented. This is often blamed on developers having a "not invented here" attitude, and wanting to build everything themselves, but I disagree (in general).

I've been a developer, and I used anything available to me as long as I understood how to use it, how it worked, and its limitations. In other words, I used third-party code/services if they were properly documented, and I could determine from that documentation that they suited my needs. If they weren't documented and I had to walk through the code (if that was even available) to figure out how it worked, then I was more likely to just rewrite it myself, on the basis that if someone couldn't write proper documentation then maybe they couldn't write proper code either. Later, when I ran a development team, I made the developers write the documentation first. They bitched about it, but we had a high level of code reuse.

There is no way to achieve SOA without reusable services, and there is no way to achieve reusable services without proper documentation of those services.

Posted by Sandy Kemsley at 11:27 AM in SOA • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Yesterday, I posted a rather whiny entry about rude customers (and Bob McIlree was kind enough to give me a comforting pat on the shoulder, virtually speaking -- thanks Bob!) so today I decided to get a bit more productive. Moving from whine to wine, I finally made my first cut of a Squidoo lens about Australian Wine in Toronto (yes, I'm a geek and this is how I spent part of my Sunday). Sort of a niche topic, true, but that's what Squidoo lenses are all about: it allows you to quickly build a one-page portal with links to other sites, Amazon products, eBay, RSS feeds, and a number of other kinds of information. Since it's all on the web, you can update it anywhere, which is why I've moved quite a bit of information about both wine and BPM from my websites to my two Squidoo lenses.

I want to add a bit of meat to this post to offset the whine of yesterday, and coincidentally (before I saw his comment), I was reading Bob's post on SOA and Data Issues and the need to maintain a source system of record (SSoR) for data. In particular, he discusses a conversation that was passed along to him from another organization:

A, the SOA implementer, argues that application-specific databases have no need to retain SSoR data at all since applications can invoke services at any time to receive data. He further opined that the SOA approach will eliminate application silos as his primary argument in the thread.

B, the applications development manager, is worried that he won't get the 'correct' value from A's services and that he has to retain what he receives from SSoRs to reconcile aggregations and calculated values at any point in time.

Since I'm usually working on customer projects that involve the intersection of legacy systems, operational databases, BPMS and analytical databases, I see this problem a lot. In addition to B's argument about getting the "correct" value, I also hear the efficiency argument, which usually manifests as "we have to replicate [source data] into [target system] because it's too slow to invoke the call to the source system at runtime". If you have to screen-scrape data from a legacy CICS screen and reformat it at every call, I might go for the argument to replicate the mainframe data into an operational database for faster access. However, if you're pulling data from an operational database and merging it with data from your BPMS, I'm going to find it harder to accept efficiency as a valid reason for replicating the data into the BPMS. I know, it's easier to do it that way, but it's just not right.

When data is replicated between systems, the notion of the SSoR, or "golden copy", of the data is often lost, the most common problem being when the replicated data is updated and never synchronized back to the original source. This is exacerbated by synchronization applications that attempt to update the source but were written by someone who didn't understand their responsibility in creating what is effectively a heterogeneous two-phase commit -- if the update on the SSoR fails, no effective action is taken to either rollback the change to the replicated data or raise a big red flag before anyone starts making further decisions based on either of the data sources. Furthermore, what if two developers each take the same approach against the same SSoR data, replicating it to application-specific databases, updating it, then trying to synchronize the changes back to the source?

I'm definitely in the A camp: services eliminate (or greatly reduce) the need to replicate data between systems, and create a much cleaner and safer data environment. In the days before services ruled the earth, you could be forgiven for that little data replication transgression. In today's SOA world, however, there are virtually no excuses to hide behind any more.

Posted by Sandy Kemsley at 10:08 PM in SOA • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

That's the thing about mashups, almost all of them are illegal

I heard that (and unfortunately am unable to credit the source) in the "scrAPI" session at Mashup Camp, in which we discussed the delicate nature of using a site that doesn't have APIs as part of a mashup. Adrian Holovaty of ChicagoCrime.org (my favourite mashup at camp) was leading part of the session, demonstrating what he had done with Chicago police crime data (the police, not having been informed in advance, called him for a little chat the day his site went live), Google maps, Yahoo! maps (used for geocoding after he was banned from the Google server for violating the terms of service) and the Chicago Journal.

Listening to Adrian and others talk about the ways to use third-party sites without their knowledge or permission really made me realize that most mashup developers are still like a bunch of kids playing in a sandbox, not realizing that they might be about to set their own shirts on fire. That's not a bad thing, just a comment on the maturity of mashups in general.

The scrAPI conversation -- a word, by the way, that's a mashup between screen scraping and API -- is something very near and dear to my heart, although in another incarnation: screen scraping from third-party (or even internal) applications inside the enterprise in order to create the type of application integration that I've been involved in for many years. In both cases, you're dealing with a third party who probably doesn't know that you exist, and doesn't care to provide an API for whatever reason. In both cases, that third party may change the screens on their whim without telling you in advance. The only advantage of doing this inside the enterprise is that the third party ususally doesn't know what you're doing, so if you are violating your terms of service, it's your own dirty little secret. Of course, the disadvantage of doing this inside the enterprise is that you're dealing with CICS screens or something equally unattractive, but the principles are the same: from a landing page, invoke a query or pass a command; navigate to subsequent pages as required; and extract data from the resultant pages.

There's some interesting ways to make all of this happen in mashups, such as using LiveHTTPHeaders to watch the traffic on the site that you want to scrape, and faking out forms by passing parameters that are not in their usual selection lists (Adrian did this with ChicagoCrime.org to pass a much larger radius to the crime stats site that its form drop-down allowed in order to pull back the entire geographic area in one shot). Like many enterprise scraping applications, site scraping applications often cache some of the data in a local database for easier access or further enrichment, aggregation, analysis or joining with other data.

In both web and enterprise cases, there's a better solution: build a layer around the non-API-enabled site/application, and provide an API to allow multiple applications to access the underlying application's data without each of them having to do site/screen scraping. Inside the enterprise, this is done by wrapping web services around legacy systems, although much of this is not happening as fast as it should be. In the mashup world, Thor Muller (of Ruby Red Labs) talked about the equivalent notion of scraping a site and providing a set of methods for other developers to use, such as Ontok's Wikipedia API.

We talked about the legality of site scraping, namely that there are no explicit rights to use the data, and the definition of fair use may or may not apply; this is what prompted the comment with which I opened this post.

In the discussion of strategic issues around site scraping, I certainly agree that site scraping indicates a demand for an API, but I'm not sure that I completely agree with the comment that site scraping forces service and data providers to build/open APIs: sure, some of them are likely just unaware that their data has any potential value to others, but there's going to be many more who either will be horrified that their data can be reused on another site without attribution, or just don't get that this is a new and important way to do business.

In my opinion, we're going to have to migrate towards a model of compensating the data/service provider for access to their content, whether it's done through site scraping or an API, in order to gain some degree of control (or at least advance notice) of changes to the site that would break the callling/scraping applications. That compensation doesn't necessarily have to mean money changing hands, but ultimately everyone is driven by what's in it for them, and needs to see some form of reward.

Update: Changed "scrapePI" to "scrAPI" (thanks, Thor).

Posted by Sandy Kemsley at 07:29 PM in SoftwareDesign • mashupcamp | Permalink | Comments (6) | TrackBacks (3) | Add to del.icio.us

The flight home from Mashup Camp was a great opportunity to catch up on my notes from the past couple of weeks, including several ideas triggered by discussions at the TIBCO seminar last week: some because I disagreed with the speakers, but some because I agreed with them. I split my opinions on the discussions about implementing BPM systems, specifically about the role of a business process analyst, and agile versus waterfall development.

First of all, the business process analyst role: Michael Melenovsky sees this as important for BPM initiatives, but I tend to feel the same way as I do about business rules analysts: just give me a good business analyst any day, and they’ll be able to cover rules, process, and whatever else is necessary for making that business-IT connection. Furthermore, he sees the BPA as a link between a BA and IT, as if we need yet another degree of separation between the business and those who are charged with implementing solutions to make business run better. Not.

There were some further discussions on how business and IT have to collaborate on BPM initiatives (duh) and share responsibility for a number of detailed design and deployment tasks, but this is true for any technology implementation. If you don’t already have a good degree of collaboration between business and IT, don’t expect it to magically appear for your BPM initiatives, but do take note that the need for it is at least as great as for any other technology implementation. How we’re supposed to collaborate more effectively by shoehorning a BPA between a BA and IT is beyond me, however.

Melenovsky also had some interesting “lesson learned” stats on the correlation between the time spent on process discovery and model construction, and the success of the BPM initiative: basically, do more work up on your analysis and up-front business-focussed design, and your project will be more successful. Gartner found that over 40% of the project time should be spent on process discovery, another 9% on functional and technical specifications, and just 12% on implementation. In my experience, you’ll spend that 40% on process discovery either up-front, or when you go back and do it over because you implemented the wrong thing due to insufficient process discovery in the first place: as usual, a case of choosing between doing it right or doing it over.

That directly speaks to the need for agile, or at least iterative, development on BPM projects. You really can’t use waterfall methods (successfully) for BPM development (or most other types of technology deployments these days), for so many reasons:

1. When implementing new (and disruptive) technology, whatever business tells IT that they want is not accurate since the business really isn’t able to visualize the entire potential solution space until they see something working.
2. While IT is off spending two years implementing the entire suite of required functionality in preparation for an all-singing, all-dancing big bang roll-out, the business requirements will change.
3. During that two years, the business is losing money, productivity and/or opportunities due to the lack of whatever BPM is going to do for them, and is building stop-gap solutions using Excel, Access, paper routing slips and prayer.
4. That all-singing, all-dancing complex BPM implementation is, by definition, more complex and therefore more rigid (less flexible) due to the amount of custom development. It makes sense that you can’t use a development methodology that’s not agile to implement processes that are agile.
5. The big bang roll-out is a popular notion with the business right up to the point when it happens, and they discover that it doesn’t meet their needs (refer to points 1 and 2). Then it becomes unpopular.

Instead, get the business requirements and process models worked out up front, then engage in iterative methods of designing, implementing and deploying the systems. Deliver “good enough” processes on the first cut, then make iterative improvements. Don’t assume that the business users aren’t capable of providing valuable feedback on required functionality: just make them do their job with the first version of the system, and they’ll give you plenty of feedback. Consider the philosophy of an optional scope contract rather than a fixed price/date/scope contract, whether you're using internal or external resources for the implementation. Where possible, put changes to the business processes and rules in the hands of the business so that they can tweak the controls themselves.

Posted by Sandy Kemsley at 01:47 PM in BPM • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

Joe McKendrick posted last week about whether open source or SOA is killing the software industry faster, right on the heels of a couple of articles in eWeek about how E-Trade is switching to open source (E-Trade's not just implementing Linux, which would hardly raise an eyebrow these days, but also components higher up in the stack, such as web server, application server and transaction management software).

From the point of view of the software industry, these are both disruptive technologies that fundamentally change the way that business is done. Funny, after all these years of introducing disruptive technologies to other businesses that resulted in some pretty major upheavals, software companies are getting it back in spades.

As for SOA and other technologies that make software development faster and easier, I say "bring it on". I have little tolerance for systems integrators (or the professional services arm of software vendors) that won't use newer, better technology when it makes them less money, although there are a few of them that seem to get it.

Posted by Sandy Kemsley at 06:43 PM in SOA • SoftwareDesign | Permalink | TrackBacks (0) | Add to del.icio.us

I received the call for papers for the 9th International Business Rules Forum, which has prompted me to browse through the other business rule-related tidbits that I've been viewing over the past few weeks. If you've been reading Column 2 for a while, you already know that I think that business rules are a crucial feature in BPM, whether the BPM contains them inherently or as an add-on: you can find some of my previous posts on BPM and business rules here, here, here and here.

Rolando Hernandez recently posted a short term outlook for business rules -- in short, that BR provide huge competitive advantage through business agility -- plus an opinion on the differences between a business analyst and a business rules analyst.

The business rules analyst is focused on separating rules from code. The rule analyst walks and talks business... The rule analyst talks about business rules and business logic. The rule analyst means business.

The business analyst sees rules as code. The business analyst talks about the system. A business analyst is often a systems analyst by nature, and by training... The systems analyst means code.

I don't think that there is a big difference in the inherent skills of business analysts and business rules analysts; rather, I think that systems analysts need to stop foisting themselves off as business analysts. Rolando starts a paragraph describing the business analyst ("the business analyst sees rules as code"), segues through an assumption ("a business analyst is often a systems analyst by nature, and by training") and by the end of the paragraph is referring to the systems analyst rather than the business analyst, as if there were no difference. Yes, this happens, but it's unfair to paint all business analysts with the same brush.

I also see the opposite problem, where a business user is designated as a business analyst, even though he (or she) has no skills or training in analysis; since he's not trained to write requirements that are both necessary and sufficient, the resulting solution will not do what the business needs it to do. Furthermore, since he's probably not up on the latest in associated technology areas, he's unlikely to think outside the box because he doesn't even know that the box exists.

The trick is to meet somewhere in the middle: a business analyst or business rules analyst needs to be focussed on the business, but be aware of the capabilities and limitations of the technology. The first job of the business (rule) analyst is to determine the business requirements, not write a functional specification for how the system might behave, as I've posted in the past. A business analyst needs training in the business area under study, but also needs training and experience in gathering requirements, analyzing business functions, optimizing business processes and documenting requirements, plus a high-level understanding of the functionality (not the technology) of any systems that might be brought to bear on a solution.

Posted by Sandy Kemsley at 08:52 PM in BRE • SoftwareDesign | Permalink | Comments (1) | TrackBacks (0) | Add to del.icio.us