Event processing is actually not new. It has been in use for many years in the financial industry. Algorithmic trading is an example. It has also been used in systems and network monitoring. However, there seems to be some misconceptions and confusion about how EDA relates to SOA, how it is different, and how it can be of use in mainstream applications.

If you've been pondering these questions, ebizQ is holding a virtual conference next week which will feature the luminaries in the event processing field. Roy Schulte, VP Gartner, will be giving a keynote presentation that explain much about what event processing is and what it can do. The Jake Frievald of iWay will be speaking about BI in an Event Driven World where he will present a generic event-driven BI framework gleaned from what customers have actually been implementing. We have an all star panel featuring David Luckham, father of CEP and Mani Chandy, professor at Cal Tech. Then Charles Brett will present Forrester's Event Processing Taxonomy.

This conference will feature a brand new virtual environment which includes a networking lounge and new virtual booths. Attendees who visit all the booths will be eligible to win a GPS system. At the end of the day we are giving away 5 copies of David Luckam's book "The Power of Events", for the five best questions of the day.

So come and ask your questions. This event is pretty much guaranteed to provide you with valuable information.

Now, this intrigued me because, if the truth be known, lately I've been taking the "I" off my ICC slides and just renaming it the Competency Center for SOA initiatives - the thought being that integration efforts are now being subsumed as a part of SOA, or as an enabler to SOA - but becoming less of a separate initiative. But John insisted that integration was indeed important in and of itself, and is being driven by enterprise strategies for aligning IT and Business, (and this includes SOA), and the need is to create formal governance processes and this demands and ICC, The second driver John mentioned is data warehousing and business intelligence, and the need to create a common view of the customer, or a 360 degree view of the business. These initiatives include master data management and integration. The third driver John mentioned is regulatory or other compliance issues, data security, and privacy, where the ICC becomes the center for maintaining data quality.

Must admit I had some difficulty getting my mind around the last one. The ICC responsible for Data Quality? Isn't that the realm of the data center, which most large organizations have had in place for years? But John insisted that federated data requires a centralized governance group to manage the canonical models and map the semantic meaning of data across business domains.

In my experience, data governance is more about politics than anything else - who owns the data and who can access it. And of course, governance needs governors. If no one is responsible for enforcing governance policies, how is governance going to actually be implemented? So I agree that these issues of control and governance of distributed and federation information require some changes to the org chart in order to make them happen. But what is the correct organization and what should the responsibilities of a CC be?

John Schmidt outlined core competencies he has defined as part of the practice:

1. Financial management. The ICC operates as a shared service. This is a set of best practices around charge back for shared infrastructure and individual services.
I think this capability is definitely needed for SOA as well.
2. Architecture . The ICC does not do enterprise architecture, but is responsible for the information architecture. They work with the enterprise architecture group, and "connects the dots", by mapping schemas to physical data sources to enable the translation, transformation, and integration. This ICC is the central federated repository.
I asked John what he thought about using semantic metadata to enable this instead of all the proprietary mapping techniques, and he responded that it's not a viable alternative today.
3. Business Process Management. According to John this is not BPM per se, but this includes service flow modeling, information flows, business event modeling, and common definition of business events.
Sounds to me this is more about SOA, than integration.
4. Integration methodology . The process of running an ICC, defining it, organizing it, all the things you need to run an integration group, and how it will interact with other IT groups.
5. Metadata management . The core tool is the metadata repository. The ICC group is responsible for data assets. Metadata ends up being a federated model. There are multiple repositories, and all have different views. The ICC understands the federated model and focuses on the key integration points between the different parts of the organization.
6. Modeling management . This includes techniques around canonical data modeling, what are the best practices, how do you build them.
7. Integration Systems. This is about running integration systems as a specific class of applications – all the discipline of how your manage, plan and operate the system. Formerly, when he was at the Bank of America, John Schmidt was responsible for running the the biggest Web methods integration system in the world. There was never any time when there were no transactions going through it. He said doing maintenance is like changing the tire while the car is moving. – how do you do maintenance. Changing tires on the care when it’s moving. Business rules, hierarchy of services. All needs to be managed. Integration will be a core competency and discipline.

In his book, Integration Competency Center: An Implementation Methodology, John defines different ICC models:

While I absolutely agree with John that organizations truly need to develop core competencies in integration, I think it is less clear what the roles and responsibilities of an ICC should be, and some of that depends on how organizations approach integration - whether it is a strategic initiative in itself, part of a an SOA strategy, or (as it is in most cases) a tactical solution for implementing a new business capability, as well as the model of the ICC or CC, or SOA CC.

While some of the roles and responsibilities John outlines may not all fall within an ICC, I think organizations that are seeking more agility through integrated solutions that cross existing application boundaries, need to think about these roles and responsibilities and define where they lie within the organization. Without governance we're going to to fall into the lawlessness of the wild west, and distributed, federated approaches will quickly run into problems. Governance requires governors.

So where do you fall on this spectrum? Are you building an all encompassing ICC? Are you creating different org structures? Are you experiencing the pain of having to make these decisions yet?

I decided to hold off on this blog post this until I actually had a chance to check this out. Lombardi is claiming it’s one of the easiest modeling tools on the market. I’ve used the SaaS modeling tools from TIBCO, and Intalio, and I’ve looked at some Visio based tools. I wanted to check into this one. So here’s my first impression.

My initial question when I logged in was “where is the BPMN pallet?” I was expecting to see all the BPMN shapes up there to drag and drop. Instead, Blueprint offers two views. One is the business level discovery view which essentially has milestones and activities (business level ideas). To add decision points and loop backs, etc, you right click on a line to choose a shape. I wondered if Lombardi had read the Israeli pilots manual. There was a story (I admit I don’t know if it’s actually urban legend because I have not verified it) that a panel tried to determine best practices by the best Israeli fighter pilots. What made them so successful? The pilots only agreed to talk after they were promised there would be no recriminations. But the answer was that when they got into a combat scenario, they shut down most of the control panel to only focus on what was relevant to finding and shooting the enemy. This is the approach that Lombardi has taken. Not a lot on the pallet to get confused by, but you can go deeper by right clicking.

In the discovery view you add things like business owners, process experts, inputs and outputs, any problems (which can also be color codes), and documentation. In other words it’s the initial business view. From there, you can simply change views to the Process Diagram and get the full BPMN diagram with swim lanes. There is also a documentation view. It is fairly easy to link to sub processes and go back and forth between linked processes. I found this to be a major differentiator from other tools I’ve tried.

Lombardi also announced packages, which are built around things you would do with Blueprint. These are service offerings that include Lombardi’s BPM methodology. They announced a Process Inventory Package, a three week service offering where they will inventory all the processes in a certain area of the business to increase productivity. There is also a Process Assessment package, a 3 day service which focuses on a certain area of the business with the goal of understanding which of the detailed processes impact key KPIs and SLAs. There is also a two-week Process Analysis to detail a BPM process. These are all fixed priced service offerings.

You can go online and check Blueprint out for free, which allows you to model a few processes. The full version will allow you to import Visio diagrams, of course model more processes, export diagrams to PowerPoint, print them, provide a shared repository, and integrate with Teamworks to automate the processes.

The major theme is Smart SOA. Drew Carey stated he had no idea what SOA is (and many in the audience around me indicated he was not alone) but IBM does it smarter. Steve Mills indicated that IBM does it smarter because of the 6550 customer implementations IBM has done. In fact the number one announcement was that IBM is the industry leader in SOA implementations. Over 272 customers speaking this week. IBM is capitalizing on and productizing this experience. Also announced was the release of 270 agility metrics 270. You can even take an SOA health check.

Another way IBM is capitalizing on the experience is by creating industry solutions. One of the solutions that has me very intrigued is Smart SOA approaches for going green. Have to wait until tomorrow to hear what that's all about.

IBM is also announcing a BPM Suite with Events Management, which includes the Apsoft acquisition. Last year IBM introduced an electronic game to teach BPM. They have introduced the curriculum into hundreds of colleges.

One of the more interesting ideas IBM is moving forward with is creating social networks to enable architects, analysts, business people, academics, to share ideas around SOA. There is a 72 hour SOA JAM going on. You can join it from your desktop.

Onto more sessions.

I recently spoke with Truviso about an innovative approach they have developed for delivering on-demand information in high data volume environments. As Roman Bukary, VP marketing and business development, explained it, instead of executing SQL queries against large volumes of data, Truviso analyzes the information as it moves through the pipe. It actually uses standard SQL, but can apply queries to high volume data streams, and feed dashboards with the information or even trigger data-driven actions and alerts.

The company was co-founded in 2005 by Berkley professor Michael Franklin and his assistant Sailesh Krishnamurthy, who is now the chief architect. They developed an engine that uses standard SQL queries to analyze data as it moves across systems, regardless of where it comes from. The result is massive scalability and performance, clocked at 100,000 records per second on a single machine. Additionally, thousands of concurrent queries can be run continuously and simultaneously on a single server, and the queries can be run over both real-time and historical data from within a single engine. Truviso uses open source database PostgreSQL which enables data to be optionally persisted for replay, back-testing, drill-down, bench-marking and other purposes. The system can be run distributed across applications, databases, and edge devices, allowing for massive linear scalability. The system includes integration components so it can accept data from multiple different sources, including message queues. Each connector is provides transformation capabilities.

Foreign currency trading was the first market Truviso entered. They also have solutions for capital markets, retail inventory, logists, SOA/Network monitoring, and RFID/Sensor Network.

Because it built on top of PostgreSQL, it can deploy natively on any OS or hardware platform. It can run in a virtualized environment, and Truviso is (or will soon be) available as a SaaS solution. Because it uses standard SQL companies can migrate historical reporting to real-time analysis in a matter of hours.

BI meets event processing. On demand BI. Intelligent event processing. This technology seems to span categories, as well as uses in the enterprise.

Sun is planning to create an IT GRC eco-system which includes ISV partnerships, SI solutions, acquisitions and other Sun product offerings. You should expect to hear more announcements starting next quarter. Below is a graphic of how Sun views a GRC eco-system.

In Sun’s view identity and access management are big components of a compliance solution, and a good place to start. Companies then can build out the rest of the capabilities through partner solutions.

Today Sun does not have anything for the management layer, but they’re working on that. I asked Nicolas where ITIL would fit, as it is becoming popular for IT Governance, and was told it would be considered one of the frameworks in the eco-system and there could be others. He stated that customers often develop their own frameworks, or begin with a standard and then customize it, and the logical view will accommodate that.

However, I was left with the question as to how SOA Governance, also a very big topic these days, fits into the IT GRC eco-system defined by Sun. SOA is changing the very nature of what we call an IT application. The monolithic applications of the past are giving way to mash-ups and composite applications. Many of the same policies that pertain to IT applications, including access control and identity management, also pertain to services. It would seem to me to be a natural conclusion that SOA Governance is a part of the overall IT GRC. How can you provide end-to-end security, risk management and compliance if the policies are not equally applied to the fast growing part of the IT portfolio represented by services? Can they really be separate disciplines, technologies and solutions? Does SOA Governance belong on the GRC roadmap? How are you handling it you your organization?

"Here was an OS that, from an enterprise IT standpoint, had almost nothing going for it: No major new technologies; no paradigm-shifting architectural changes; nothing to whet a system administrator's appetite. What it did have was layers and layers of consumer-focused baggage: Pervasive DRM plumbing; dubious multimedia prioritization tweaks; OS X-envy driven eye candy. Basically, it was an OS designed to secure Microsoft's seat at the RIAA/MPAA roundtable, and little else."

Here was the eye opener. Vista represents little more than an expense for IT departments - it has nothing else to offer. But I must admit, even after having to spend more than I would have liked to upgrade software which was admittedly old but still did the job I needed it to do, I like Vista. The user experience is a big improvement over XP. Eye candy is part of the appeal. But it all seems to work more smoothly. It's easier to use. There's less arm wrestling with MS Word and Powerpoint. Things are more integrated. It feels more productive. Now from an IT enterprise point of view I have to admit it's probably not enough to justify the expense. But it made me understand the appeal of the Mac. My son-in-law asked for the new Mac OS for his birthday. Can you imagine any PC user wanting the gift of an operating system for their birthday? Usually I'm squarely on the side of enterprise IT, but I'm starting to change my tune a bit. The end user experience should not be discounted as unimportant eye candy. IT needs to enable a smooth and seamless user experience. It cannot be a mere secondary afterthought to other IT requirements.

Which brings me to my musings on Web 2.0. On Mar. 19th ebizQ held it's first Web 2.0 virtual event, a live Roundtable Discussion on Web 2.0 and SOA (you can view the archive). We had an all-star panel including Dion Hinchcliffe, Ron Schmelzer and Doug Wilson, and they were all great at explaining the role and benefits of Web 2.0 in the enterprise. But the audience questions were equally interesting. One attendee stated that while blogging was successful in the enterprise, they experienced problems with viruses from social networking sites such as Facebook, even after deploying all the virus, adware and malware protection, and wondered what the ROI was for the organization. Another attendee wondered if spending time on social networking sites meant lost productivity of employees. Dion Hinchcliffe said they have reviewed all the case study data and there is definitely an ROI for Web 2.0. Doug Wilson of IBM stated that internally deployed social networking software has lead to increased productivity across the organization. He made that case that many business services are executed by humans who need to interface with other humans as well as systems to do their job. Web 2.0 provides the support to carry out human based business services which in turn rely on back end system services.

So the whole idea of how humans interact with computers is becoming more important in the enterprise. Interestingly, a poll of attendees showed that 50% of the live online audience were investigating Web 2.0 solutions, 13% were piloting them, 38% had Web 2.0 solution(s) in production, but 0% said that Web 2.0 was prevalent throughout the organization. However, Doug Wilson indicated that the social networking solution internal to IBM is largely responsible for enabling the the highly matrixed reporting structure and enabling people to connect more easily.

Bottom line - Web 2.0, while difficult to precisely define, is here, growing, and likely to change our expectations of the way we work. In the future, we are going to expect a seamless experience. We are going to be less tolerant of being forced to do things the way IT works, we are going to expect IT to enable the way we want to work. So shame on you Microsoft for creating an OS that enterprise IT doesn't need but end users do. And shame on all the IT folks dismissing Vista as DOA and not considering for a moment the benefits of the human interface advancements as being valid. There's a lesson to be learned on both sides.

Now I think I'll go tag something.

However, I absolutely could not fail to tell you about a great virtual event that will happen on Wednesday, March 19th. As part of the ebizQ In Action Virtual Conference Series there will be a LIVE Roundtable discussion on Web 2.0 and SOA. We have an amazing panel which includes: Dion Hinchcliffe, Founder and Chief Technology Officer for the Enterprise Web 2.0 advisory and consulting firm Hinchcliffe & Company, creator of Web 2.0 University, and co-author of "Web 2.0 Patterns: What entrepreneurs and information architects need to know"; Ronald Schmelzer, Managing Partner at ZapThink, well-known SOA expert, lead author of "XML And Web Services Unleashed", and co-author of "Service-Orient or Be Doomed"; and Doug Wilson is an IBM Distinguished Engineer, Chief Technology Officer for Portal & Collaboration Products, and the Vice Chairman of the Software Group Architecture Board for IBM Software Group.

With this group of esteemed panelists I can guarantee you a lively and thoughtful discussion. If you miss the live broadcast, it will be available in the archive. However, you must attend the live session to win any of the giveaways. ZapThink is giving away one free pass to it's Practical SOA course March 25, 2008, Newark, N.J. ZapThink is also offering ebizQ members $100 of the regular $295 price. Use this link for ebizQ members and enter the promo code EBIZQPSOA.

Hinchcliffe & Company is offering a free course in their Web 2.0 University designed to teach Web 2.0 and Enterprise 2.0 techniques on May 6-7, New York, NY. They are also offering $300 discount of the regular $995 price. ebizQ members can use this link for the discount.

But wait, there's more (about now I feel like Ellen DeGeneres who gives all her audience members gifts), but you've got to attend the live broadcast for this one. ebizQ will award 10 copies of "Web 2.0 Heroes" by Bradley L. Jones. Due out next month, the book profiles innovators at companies like eBay, Microsoft, Sun Microsystems, Adobe/ Macromedia, YouSendIt, IBM,Bloglines, Ning, Technorati, Zoho and many others to show where Web 2.0-based business is going in the future.

I kid you not. This LIVE event is not to be missed. Register now.

TIBCO’s ActiveMatrix allows companies to design, develop, deploy, manage and govern SOA solutions without having to worry about each of the underlying technologies. It is a grid architecture that enables different technologies to be “plugged in”. The grid provides the integration and common management and governance across all the technologies.

In ActiveMatrix 2.0 BusinessWorks now runs natively as a container. You can take existing BusinessWorks projects, including BPEL orchestration, can compose it with java or .Net and deploy and manage it all as a single application. Adapters also run in service containers. You can take a configuration that defines what functionality is being exposed from the application, such as SAP or a database, deploy it in an ActiveMatrix container, and it becomes a fully managed service.

ActiveMatrix 2.0 has added a new standalone engine for service mediation. Tibco has unbundled the service bus (ESB) from BusinesWorks to provide a lower price entry point for developing new services that can be plugged into the ActiveMatrix platform.

TIBCO has also expanded its SCA support. SCA emphasizes the decoupling of service implementation and service assembly from the details of the technical infrastructure and access methods used to invoke services. SCA components operate at a business level and use a minimum of middleware APIs. The companies contributing to the SCA standard include: BEA Systems, Cape Clear Software, IBM, Interface21, IONA Technologies PLC, Oracle, Primeton Technologies Ltd, Progress Software, Red Hat Inc., Rogue Wave Software, SAP AG, Siebel Systems, Software AG, Sun Microsystems, Sybase, TIBCO Software Inc. As more companies build SCA support into their platforms it will make services easier to deploy. This is TIBCO’s goal is supporting SCA.

ActiveMatrix 2.0’s SCA support includes a composition editor as well as expanded deployment and management support. It provides an integrated service view which shows a service with all its dependencies. This enables operations to more easily understand what might be the root cause of a failure. Policy management is also available for all services. When services are built with ActiveMatrix all the SCA annotations are automatically generated. TIBCO claims this contributes to up to 50 percent greater productivity and lower cost of ownership. Integration, maintenance and governance are built into the platform, and available regardless of the technology used to develop the service.

ActiveMatrix 2.0 can also import process models developed in Visio or Aris, make them live, and tie them to the applications. The integrated registry and repository can then show how business processes are impacted by changes. This allows better alignment between IT and the Business.

From a pure architectural standpoint, ActiveMatrix 2.0 represents best practices for creating an integrated infrastructure. It simplifies SOA by providing a set of common infrastructure services to all business level services, regardless of technology difference, making it almost as easy to deploy a new business service as it is to plug in a new appliance into the electrical grid.

According to Avrami Tzur, VP SOA, HP’s customers were asking for help in determining the governance process and deciding whether to focus on quality or performance. In response HP has launched three new professional services, all of which are short in length, with defined deliverables. The SOA Validation service comes with Systinet and the best practice governance process for validating services embedded in it. This can then can be expanded across the organization. The Quality service includes pre-defined metrics and processes. Because so many customers already have the Mercury quality software this service focuses on validating the quality and performance. The other new service regards helping customers with performance of their SOA services.

The software announcements include a new release of Systinet SOA Manager. This includes the Talking Blocks software purchased by HP 3-4 years ago. It includes runtime management, security, and routing mediation of web services. HP’s vision is to provide software to govern the whole life cycle of a service.

HP also announces a new product. HP SOA Registry Foundation packages and simplifies the Systinet registry, so it can now ship to many OEMs. It can easily be put on developers’ desks, and into distributed replicated environments such as oil and gas. The US Army is putting it in tanks. The registry is autonomous but is also integrated with Systinet and supports interoperability at runtime to achieve the full governance solution.

HP is also offering a new suite of SOA education and training courses. Last May HP announced its SOA strategy as focusing on governance and management - two areas where HP has considerable enterprise experience and software offerings. HP's strategy is to remain platform agnostic and provide the services and software to manage, govern and provide quality assurance for services. This announcement of new services and software furthers this vision.

Like all of the ebizQ polls and surveys have shown, the Amberpoint survey found that the majority is still in the early stages of adoption. But there were also some surprising results. The Amberpoint survey showed that relatively few (20%) of the deployed SOA solutions are standalone, single department systems. Most are used across departments or even externally. This survey indicated that the biggest benefit of SOA is its used in addressing integration issues. This, again, is an indication of the early stages of the market as few companies have mastered the art of building reusable business services and achieving true business agility based on SOA infrastructures.

But IMHO the singularly most surprising finding of this survey is that 98.5% of respondents called their SOA implementations "successful". When in the industry have we EVER seen a 98.% success rate with any technology or approach - especially in an early market? I think this is something worth exploring more. What has been your experience. Have you had a 98.% success rate?

Download a copy of the survey results.

Listen to the podcast with Ed Horst, VP Marketing, Amberpoint  

Download file

This week I spoke with James de Raeve, VP of Certification for The Open Group. The Open Group is kicking off its IT Specialist Certification program to attendees at the conference along with current Open Group members. We talked about the role of the IT Specialist, and the new certification program.

You can listen to the Podcast here:

Listen to the entire podcast Download file

You can also read the transcript of the podcast.

Download file

Listen to or download the podcast below: