Welcome to a full transcript of the keynote Webinar from our recent SOA in Action" Virtual Conference. Parcipants in this Webinar include Beth Gold-Bernstein (BGB) and Roy Schulte (RS); Jake Freivald of iWay Software fielded questions during the Q&A session.

BGB: Hello, everyone! And welcome to the second ebizQ "SOA in Action" Virtual Conference. I'm Beth Gold-Bernstein, chair of the conference. Before getting started, let's take a moment to highlight the features and functions of our trade show. The environment is highly interactive. You can check with company reps and other visitors, send a business card, or leave a message.

This conference includes virtual booths. You can visit the booths to win prizes, download literature and interact with company reps and other visitors. After today's presentation, we will be taking your live questions. To submit a question, click on the "ask a question" button on the grey console. To download a copy of the presentation, click the "files" button. To enlarge any of the slides, click on the magnifying glass to the right of the slide.

And, now, it's my great pleasure to introduce Roy Schulte, VP of Gartner, who is going to lead off this conference by talking about the state of the art in SOA. We would like to thank iWay Software for sponsoring this keynote presentation. After Roy talks, Jake Freivald, VP of corporate marketing of iWay Software will be getting on just to say a few words.

RS:Thanks, Beth. It's always fun to work with you on these webinars. So I've been asked to talk about what is new in service-oriented architecture. SOA has been around for some time now -- eleven years to be precise. And I think by now most people have a pretty good sense of what it's all about. So what I want to do today in this session is to talk about the four trends that you may not know about, but things that you should know about SOA right now.

Of course, the ideas behind SOA were not new. The principles had been evolving for years in the RPC and the CORBA communities. The idea of SOA was so logical that, frankly, we thought it would catch on quickly. If you look down at the lower left of this chart, you will see that we had projected that more than a third of all large new business applications would use service-oriented architecture by 2001.

Well, that wasn't quite right. We were off by about three or four years. It took until maybe 2004 or so before a third of large applications had some traces of SOA in them. The majority of the applications today now use some service-oriented architecture in some part of the operation but many systems only use SOA in a superficial manner or some, of course, still don't use it at all.

SOA is growing, but it takes a lot longer for these things to catch on than you might expect. The basic principles of service-oriented architecture were clear way back in 1996. And they haven't really changed since then. Leading edge development teams started using SOA even before 1996, although it certainly was not mainstream and SOA had a lot of limitations to it.

SOA back then used the principles of modularity. Those modules were distributable. They could run on separate computers. The modules had formal interfaces to them. And those formal interfaces were separate from the implementation. So you could swap out one set of code, or data, and replace it with another, as long as the interface wasn't changed.

And, finally -- SOA applications have always been shareable. However, back in 1996, there were no standards of Middleware that all of the vendors would subscribe to and so we had implementations on top of message-oriented Middleware. We had others on top of CORBA, which was a standard that some but not all vendors would use. And various other Middleware mechanisms.

Because of this difference, interoperability was difficult across different operating systems and different application servers. In 1996, the notion of business process management was really part of the discussion. Generally, if you were worried about business process management or automating the flow of control, people had to do that by hand, by coding it into the application itself. So we had SOA in 1996, but it certainly wasn't that common and it had a lot of limitations to it.

In today's presentation, looking at what's different today in 2007 -- the four biggest changes that have come to SOA during the past eleven years are the ones that are listed here. The first one was the introduction of standards in the late 1990s and early 2000s. This helped make interoperability across heterogeneous platforms a bit more practical. Although, we will talk about some of those limitations that continue to bedevil us even today.

The second major point is about the expansion of SOA design patterns. Back in 1996, almost all of service-oriented architecture was done with a request/reply paradigm. That is, a SOA consumer, the consuming piece of the application invoked a service in a RPC style. This is still very common, and it's very important, but now we also have two other models of interaction that are used sometimes. One is representation state transfer or REST. And the other one is event-driven architecture. So now we have three vastly different options for SOA architecture rather than just one. They are all service-oriented architecture but they have a different mindset.

The third major change is the dramatic acceleration in the business requirements for situational awareness. Business people more and more want near-real time visibility into their company and into the external environment. So that they can sense and respond more quickly. We're seeing business dashboards in specific and BAM in general really taking off. Service-oriented architecture and event-driven architecture provide a foundation that makes BAM more practical and more common.

Last but not least, we're going to take a look at business process management and the general notion of mediation. Again, in plain SOA back in 1996, the interactions between the SOA consumer portions of the applications and the service provider portions of the applications tended to be simple and direct. Today, we see mediation being used. So those interactions are much more sophisticated and complex. And business process management is probably one of the most advanced aspects of that mediation.

Just as an aside, these four topics that we are talking about here today are taking from materials that we're preparing for an upcoming Gartner conference. So, when we are done, if you want a more complete understanding of these issues, come on out to Las Vegas for three days in early December and you can get a deeper dive into these four topics and other topics that are relevant to the same area.

So let's go to the four changes in service-oriented architecture in more detail. Let's start with the issue of interoperability. From the very start, a key promise of SOA is that you would be able to have heterogeneity. That is, a distributed application where some of the pieces were running on different operating systems or different application servers than the other pieces. Some components might be Windows. Others might be on UNIX or LINUX. Others could be on mainframes and so forth.

So you should be able to have these various components of the overall system run on different platforms but all be able to talk together, using defined interfaces and the SOA design pattern. But this requires that there is agreement on the technology for the interfaces and it requires that you have common identifiers for how a consumer can point out which service provider it needs, common formats and protocols so that the pieces can talk to each other.

SOA is easiest, of course, when you run everything within a single vendor's environment. So if we look on this chart, Business Unit One -- if you have the operating system, the Middleware, the XML http stack, the enterprise service bus, whatever communication mechanism you're using, and the application server all coming from the same vendor, then the components speak very easily to each other.

In this case, we have service consumer A invoking service B and the interaction doesn't require any gateways and is very simple to construct. But with SOA, we want that service consumer A to also be able to invoke services on other platforms. Perhaps another business unit. We want consumer A to be able to talk to service provider C. Now in that case, the host environment for service provider C has to support something that is compatible in terms of identifier, format and protocol with the service consumer A or else the communication can't take place directly. You would have to give him the gateways and transformations and so forth, if you're trying to cross these boundaries, if you don't have common formats and protocols.

So, service-oriented architecture really took off when those kinds of standards appeared. In the late 1990s, the Web came into common use and then in 2001, we had the advent of Web services. It was really the introduction of SOAP and WSDL in 2001, that sparked the widespread interest in SOA among mainstream developers. No longer was SOA limited to leading edge accounts. For the first time, in 2001, we had major vendors like Microsoft and IBM agreeing to implement the same identifiers, formats and protocols.

These standards really work, but only to a certain point. The point at which they work very well is defined by WSI, Web services Interoperability basic profile one. It's a robust standard that is implemented the same way by all the major vendors and it makes it practical to have components on heterogeneous platforms talking to each other, with little or no special coding. The basic Web services stacks and the enterprise service bus Middleware from all the major vendors talk using these formats and protocols.

If you want an example of this, go up to the IBM developer Web site and look at the WebSphere trade 6.1 (11:30) sample SOA application. And then go over to the Microsoft developer site for the dot net stock trader sample application, which is the same application but implemented on .NET instead of implemented on Java. What you can see> is a great demonstration of pretty much flawless interoperability between clients and servers on these two different platforms, able to talk to the clients or server on the other platform.

If you look at the standards on this chart, you'll see that already today, we have areas where we would rate the compatibility between vendors of a green. A green is a go, they really work compatibly. However, what you may also notice is that this only applies to certain aspects of communication. It implies the basic Web services and security, but not really to advance security or some of the administrative and provisioning standards. And you may also notice that between today and 2011, we're not anticipating a lot of growth.

The services that work today, the compatibility today, is limited to fairly low-level interactions. The kind of interactions that you would see taking place between the presentation layer of the application and the business logic layer of the application. In most cases, we're talking about a request and reply model, and we're not talking about advanced quality of service features.

Where we don't have much in the way of standards today, is for the server-to-server kind of traffic. Where a service provider component talks to another service provider component. This is where you would have a component that's doing business logic and data logic, acting as a consumer, talking to another component that's doing business logic and data logic. So these kinds of interactions, you need other types of features. Things like, quality of service -- involving perhaps message queuing. Publish and subscribe. And other kinds of capabilities.

Now, here, we have a problem in the area of SOA because the standards are not moving along very quickly. In fact, we've seen very little movement in some of these areas for years. It's taken us almost five years to get reliable messaging implemented by most of the vendors, and it's not quite here yet, although we are expecting that to happen between today and 2011.

But if you look down the rest of this chart, you will see many areas where there is not communication capability today and we don't expect any, even by 2011. So what's happened here is that the interoperability standards have slowed down significantly. It means that interactions that require asynchronous behavior like queuing or even today, publish and subscribe, can't be done using industry standards. So, we're still stuck using gateways or proprietary messaging systems.

Now, this doesn't stop SOA entirely. SOA still works within a homogeneous environment where all the platforms come from one vendor, and it doesn't stop it in a heterogeneous environment if you use the gateways. However, it does mean that the whole goal and the whole notion of a heterogeneous SOA is not coming to fruition within the planning horizon, and so there's some limitations in terms of what we can get from SOA and what we were hoping to get.

Okay, now let's turn to the second major change in service-oriented architecture and that's the broadening of its fundamental interaction patterns. As I said before, the original implementation of service-oriented architecture was all about request/reply. In fact, even today that's really what most people think of when they think of service-oriented architecture. A consumer or component delegating some work to a service provider component and then invoking a function on that service provider component using a RPC type of model. Now, there are several flavors of how the RPC is done, some is a tightly-coupled RPC, others are a loosely-coupled Web services RPC, or an even looser-coupler services document style connection -- but at the end of the day, most SOA historically has been a request/reply communication pattern.

When the Web came along, different models interaction was introduced based on XML and http. And this is representational state transfer, or REST. This is still a type of client server architecture but with this architecture, it is easy to build client modules independently of the server modules because the interactions are more standardized and more loosely coupled.

In this case, you don't create your own custom method name the way you would do in a classic method centric SOA environment. In this case, instead, the methods are defined by http. They are a get, put, post and delete. This kind of application is more pluggable and easier to modify than a standard SOA application and it works especially well for read-only kinds of work where you're pulling data from many different places on the Web.

In some circumstances, this really shouldn't be considered a type of SOA if you're just using it for plain Web browsing, for example. However, if you are using it to build an application system where you do have consumer components and service components talking to each other using the REST model, you can build a true SOA application by documenting the interfaces formally and in that case, the application would be REST but will also conform to the principles of service-oriented architecture.

For some aspects of certain types of applications, REST is a good model of interaction. But there is also a third model of interaction that is part of service-oriented architecture today and that is, event-driven architecture, EDA. In an event-driven architecture, much of the way of thinking about the application changes again. The communication is generally a one-way event delivery, it's an asynchronous message delivery. This can be point-to-point but more likely it's going to be published and subscribed.

Rather than one-to-one with both classic method-centric SOA and REST are, an event-driven architecture can be one to many or many to one, or many to many. In its transfer of information. Now, good architects all along knew that SOA needed this kind of option for certain types of work. However, generally, people who want to do asynchronous behavior and event-driven architecture didn't do it as part of their SOA environment. They had a separate parallel implementation that co-existed with SOA that implemented this on different Middleware and using different programming models.

What's changed in 2007 is that people are now implementing EDA in conjunction with the rest of their SOA. EDA is really considered part of SOA. We can use the same Middleware, the same enterprise service bus and many of the same standards like XML and XST and XSLT, in some cases, even SOAP to implement EDA as we do classic method-centric SOA. So what's happened here is we've had a fundamental growth and expansion in the programming models associated with service-oriented architecture. This allows us much more richness and power in how we build application systems.

Now, we really don't have time in today's forum to talk about the details of how REST is implemented, for example, or how EDA is implemented. But I'd thought I would throw in a quick sample here just to show you how different the method of thinking is. REST is still a type of client server architecture but in this case in REST, you don't create your own custom method name, you use the four standard http verbs. The implementation of qualities and service such as reliable messaging is done much differently. You probably can't read it on this chart or you may not be able to, but on the left side you will see an example of how you would implement acknowledgement of a message delivery using Web services. And on the right, we’re showing a REST type implementation of the same kind of feature.

Similarly, event-driven architecture differs in a number of ways from method-centric architecture. In an event-driven architecture, the events are always pushed, not pulled. It's a one-way instead of a two-way communication. The event consumers, the bit of software that receives the events, act when the event arrives, not when a request is made or on any pre-planned schedule.

And, finally, there really is no application level operation name when you're doing event-driven architecture. The only verbs that are there are mechanical verbs to send and receive the event. So therefore, the destination, the consumer decides what the operation is going to perform. It's not dictated by the sender of the event object. The idea of an event-driven architecture is really a different kind of thinking here.

It really is an example of where you might have left-brain thinking and right-brain thinking and we definitely need a third type of brain thinking because we're talking about method-centric and REST and EDA all coexisting as various programming models under the same umbrella as service-oriented architecture. The discussion about event-driven architecture naturally leads to our third topic, which is the dramatic growth in business activity monitoring.

Business activity monitoring, BAM, is one of the biggest motivators of SOA and EDA projects today. The reason why is because business people are looking to get better visibility into how their business is running. They want to know what is happening up-to-the-minute both within their company and outside of their company, with their customers, their competitors, with regulatory bodies, with the markets and even the weather. This is all about situational awareness, knowing what's going on so you can make better decisions and faster decisions. BAM, sometimes called operational business intelligence, whereas traditional business intelligence was historical, looking at data from the past, stored in data bases, BAM concentrates on looking at current events, reports of what's been happening in the last few minutes or the last few seconds.

When people talk about business activity monitoring, they generally conjure up images of business dashboards on Web pages. And, indeed, that is the most common vehicle for giving people alerts as to what's happening. However, business dashboards are not the only mechanism to deliver up-to-the-minute data to end users. There are many different mechanisms, such as email, such as paging, such as phone calls and so forth.

BAM has always relied on processing, so naturally it makes good use of event-driven architecture. But when you look at the implementation techniques in detail, there are a number of variations on the BAM design patterns. BAM doesn't always necessarily require service-oriented architecture but it usually does use it and in particular uses the EDA style of service-oriented architecture.

Unlike other types of service-oriented architecture, other uses of service-oriented architecture, BAM is directly visible to end users. A business manager can sit down and see a dashboard using the key performance indicators that they really care about. So, in that way, a BAM dashboard provides much more tangible benefits to the end users than most of the types of uses that are built on top of SOA or REST or EDA. This is really why many companies start the process of building an application by a desire to have BAM and from that, they are led to implement other things like the service-oriented architecture, REST and even business process management.

Finally, let's spend a few minutes on the fourth and last change in service-oriented architecture that has occurred in the past two years. And this is the spread of mediation services. Now, the original vision of SOA conceived of plain, direct communication between a consumer and a service provider component. That implementation could have been using any of the design patterns we've talked about here. The method-centric SOA, REST or event-driven architecture. On our picture here, we're showing the service consumers as being rectangles and the service provider components as being little circles.

The service consumers often were the presentation portion of the application, something that was directing the Web interactions with the end users or perhaps something that is directing interaction with other application systems. And the service provider components could be nested one or many layers deep and they provided the business logic and the data access logic.

Now, this simple vision of SOA works as is and is, in fact, still the most common way of implementing service-oriented architecture. It's the radant for simple applicaitons, particular applications that don't have to change very often. However, more and more, we are seeing the implementation of SOA using mediation layers. A lot of SOA is just not that small or simple anymore. SOA is inherently organic and heterogeneous in its nature. You start your applications small and you grow it. You add additional service consumers, you add additional service providers.

In many cases, the new modules that you're adding are somewhat heterogeneous. They are coming from different sources. The different sources may be using different technologies, different platforms like different operating systems, different application servers. More importantly, the designers are probably using different information models. Different data models. Different process models and so forth. Therefore, it's not easy to make these modules connect directly into modules that are written by other groups.

There is a need for services such as transformation, content-based routing, monitoring, and business process management. To help provide the grease that helps make the various component fit together. So there is a strong undercurrent among leading application developers that is leading them to change the way they build SOA systems. They are introducing these ns, these mediation type of services into the communication traffic between the service consumers and the service providers.

These mediation services are not implemented in the sender or the receiver. They are not implemented directly inside the consumer or the service provider. Rather, they are built in the middle. They are built into proxy servers, which are generally hubs of some sort, or wrappers or somehow they are built into a communication stack bound into the endpoint consumer or service provider application components. By the way, mediation is used almost all the time when you have a REST application or an event-driven architecture application. It is sometimes also used even in a request/reply application such as when you are doing a composite application on request/reply.

Now, the relationship between BPM and SOA has been often discussed. So I won't belabor that part of the issue here. In fact, some people would be really emphatic about it and say that BPM is so important that SOA is not worth doing at all if you don't do BPM. Now, most people don't go that far but certainly everyone would at least agree that BPM works well with SOA and vice versa.

But here's what you should know about business process management: it's not one thing, it's actually a general paradigm and it can and should be applied at least on four different levels. The first level of applying control of flow is a microflow. This is what happens inside of a single atomic SOA element. For example, a single service provider element may have within it, multiple different software components.

Within those software components, there are microflows as these components talk to each other. These are generally short and fast, often they are transactional, generally take place within the same computer. Often within the same process space. So this is flow management, but it's not really what most people would call business process management because it's so fine-grained. What comes out of this, however, is a service provider component which then can be composed into larger services.

The second level of flow management is really service composition. This is where you take a number of those individual atomic services and connect them together in a flow, perhaps using techniques such as business process management, built on BPEL, business process execution language. Now this level of flow control happens within a few seconds, give or take. Most people would, in fact, call this business process management. However, it's a certain type of BPM that it is generally fairly short-living and in many cases, the flow may appear to be synchronous from the outside.

But we're not done. Services that have been composed of smaller services are themselves composed together into larger, multistep processes. The multistep processes may be coordinated by a business process management engine although they don't have to be. In this third level, we are dealing with more asynchronous behavior than we did in the first two levels. The execution of a straight-through process may take minutes, hours, or even -- in some cases -- days. The flow of this process is called straight-through processing, if there is no need for a human environment, involvement, I should say.

Composite services are themselves just building blocks in larger, long-running processes. The long-running processes may last minutes, hours, or even days. And they are generally synchronous in nature. The flow of these processes can be done through manual mechanisms or they can be controlled by a business process management engine. When this process, this multi-set process has no involvement of humans -- that is, all the steps are done by a machine, it's a straight-through or STP.

However, in some cases, we do have steps that are executed by humans, manual steps. So the fourth variation on business process management is where you have a long running process that involves a combination of automated steps and human steps. And this is generally called workflow, although in some cases, people would just call this again, business process management.

So, although it's pretty widely known that business process management and SOA are related, what you need to know is that business process management is one thing. To have a really good handle on controlling the flow of execution within a SOA application or process, you need to design on all four levels of flow management. You need to work on the microflow if it happened within a single service element. You need to know how individual service elements can be combined into larger services through service composition, often using BPEL. You need to know how those composed services may just be individual steps in a long-running process that may be straight-through or that may have human involvement.

So that concludes our overview of four important trends in service-oriented architecture. As you can see, service-oriented architecture is significantly more sophisticated and more powerful than the original implementation of SOA more than eleven years ago. I've summarized the highlights of these four changes to you today. Again, if you want more information on this, you can come out to Las Vegas in December for the Gartner conference. And, also -- I will take questions on this later in the program. So with that, I will turn the microphone back to you, Beth.

BGB:Thank you very much, Roy, for that overview. Very helpful.

To hear a complete replay of this Webinar -- including slides and questions, click here.