Governance in SOA has been the subject of much discussion -- a trend some analysts say is a sign of the technology's movement to a more mature phase. And Web 2.0 is another interesting trend that opens up SOA possibilities.

Kelly Emo, an SOA product marketing manager with HP Software, has some particular interest in the state of SOA governance -- and she'll be speaking on this issue in a presentation called "Enterprise Mash-Ups for Wall Street: Leveraging SOA Web 2.0" at the Web Services SOA on Wall Street show set for February 11.

"I just see it’s such an exciting and dynamic area right now," she says.

Two Key Waves for IT

Emo predicts that two major waves are "coming to shore for IT" right now. The first is SOA, with surveys putting adoption rates at 40 to 60 percent of enterprises having SOA in a pilot phase or business deployment. She also cites a few surveys that indicate over 70 percent of CIOs have SOA plans for 2008.

"It definitely hit the mainstream in terms of thinking about it and planning for it," she comments.

The other wave is Web 2.0, which makes for an interesting time for IT. IT departments are dealing with "meaty back office problems" but end users are sometimes too impatient to wait for IT departments and their planning processes, and so they take their needs into their own hands with new approaches like mash-ups and Web collaboration.

Hero or Zero

At this point, Emo jokes that IT has the opportunity to become the hero or the zero based on whether it chooses to embrace this capability. If they do not, then they are more likely to run into issues. Emo hopes instead that IT will embrace the new "Wild West of Web 2" but pay attention to governance.

Implicit Governance

Emo thinks there's a desire for back office environments to become service oriented -- enabling the infrastructure to be governed and allowing parts to be exposed in mashable services, and in order to create these applications, IT needs a sense of control.

"Even though they can’t predict how many consumers they’re going to have out there in the Web 2 world, they can know how many they’re able to support and they manage as the load comes in when they need to allocate more resources," she explains.

Emo calls that concept implicit governance. Consumers are not aware of the governance process but are assured of getting the services they expect, while IT is also able to get ahead of the curve and be ready to offer support.

How to Orchestrate Governance

Emo has some examples for how this works in action. Say, for example, a company has back office applications that support the sales department and wants the ability to expose those to the sales department in order to create situational applications.

That company can put in technology to the back end to support security and performance along with testing capabilities that feed into the governance process, then determine the kinds of loads that the systems will support. Then, when exposing services as mashables, they place a contract between the consumers in WYSIWYG style.

"What IT can now do, because they’ve got this governance platform in place, is they can track proactively how many consumption events are taking place and what kind of traffic they’re seeing and they can get ahead of the curve," she says.

Companies could also do things like track traffic to create policies to deprovision customers that aren't using the system in order to free up capability for others.

"Think about if you wanted to allow new channels for your products or services and creative mash-ups to deliver information about your products into whole new markets where they’ll want to place orders," she says. "

I mean, that could just explode worldwide."

The company name certainly catches one's eye -- and that's partly why Wowza Media Systems chose it.

"We want to get the 'wow' into video, not only from an eye popping perspective where you see great video but also from a price and performance perspective," says David Stubenvoll, CEO and co-founder of Wowza. "I’ve been speaking with someone and when I’ve mentioned our latest pricing, they’ve responded 'WOW – za!' so it’s a perfect name."

About Wowza

Wowza launched its flagship Wowza Media Server Pro nearly a year ago, and since then the company has racked up more than 4,400 global licenses and won awards. Customers include everyone from content delivery networks (CDNs) to service providers, and even the Tampa Bay Buccaneers. Department store giant Macy's is also using the product as a part of its internal training functions.

But Wowza's product was actually created as an accident. Stubenvoll and co-founder Charlie Good left Adobe and began a simple video blogging solution that wrapped Flash Media Server with a WordPress core and ran into problems when the server wouldn't work as they wanted. They also were unhappy with the price. End result: they wrote their own server.

"While our initial service didn’t exactly rock the world, people started asking us about our server and we had our first sale of the server in 20 minutes of a first phone call," Stubenvoll muses.

After re-architecting the system with an industrial strength product in mind, Wowza Media Server was born -- the only independent media server company out there, Stubenvoll points out. Future plans are to offer customers video capabilities that will go to any device on any platform with any video codec -- and this is all with a piece of code only about a megabyte in size.

Service Providers Using Wowza

One company currently using Wowza is Interroute, a large fiber owner in Europe. Interoute does VoIP but also CDN and similar services. Yonden Media Works, StreamGuys, and Nacamar are a few other providers using the service.

Wowza also has a product for Amazon Elastic Compute Cloud, which is like a hybrid type of SaaS -- the company licenses the software along with hosting services and bandwidth at a price paid to Amazon. Customers using Wowza Pro through the program can basically use the same functionality as those using Wowza in an on-premise installation.

"The great thing and the compelling thing about these EZ2 platform is that it allows you to easily add and delete servers so you can expand and contract as your needs require and the bandwidth," he points out.

Competition with Adobe and Microsoft

Stubenvoll says Wowza sees itself as the agnostic player in the media server market.

"Flash is tremendous. Silverlight is great. Java is wonderful. And, as we’re getting towards this notion of convergence, I want to see my Heroes whether it’s through my living room or on my desktop or on my mobile phone," he says.

In today's world with disparate systems and media servers that serve the client, publishers need to release disparate systems to meet needs, but Stubenvoll says that Wowza will make it so that publishers can use the same servers for all of those outlets. Wowza can facilitate combination with software from Adobe and support phones powered by Java running Silverlight.

"In the end, I think we are going to be a boost to each of these systems and I would say our relationship with these companies today is cordial," he asserts.

Executive Summary by Krissi Danielsson 

Web 2.0 applications offer a great way for businesses to boost agility by finding new and more interesting ways to reach customers -- but without a clear and well thought-out strategy, Web 2.0 can easily become a Mess 2.0.

Prakash Sinha, a product manager with Cisco, recently joined ebizQ to discuss the strategies he shared in a webinar entitled How the Internet's Largest Bookstore Mashes SOA and Web 2.0.

Where to Develop Web 2.0 Applications

According to Sinha, the network is actually the most logical place to develop Web 2.0 apps -- and he cited Netscape and Google as proof.

Both were both key companies in changing the way user-facing applications are written; as Sinha notes about Google's introduction of Ajax.

"What these technologies did was take all the applications from the data center and web-enabled them," he says. "And now, with the Google revolution, it’s actually creating these rich Internet applications based on these Ajax and JavaScript object innovation technologies."

Mashups and remix applications are another example. These applications create traffic on the network and demonstrate the need to scale and secure the infrastructure. Layer 4 through layer 7 are changing, and the network is also changing, to become more application-aware and application fluent, he points out.

Proof of Concept for Web 2.0

Sinha cites another case example in a Cisco customer that offers the largest bookstore in the world and had fulfillment channels that it wanted to reuse. The customer had a large competitor based in the West Coast and needed to reduce its cost of doing business, so the customer ended up creating new storefronts on its infrastructure, as well as partner storefronts. In order to mediate challenges, such as not being able to control security for partners, the customer used Cisco technology to meet these needs and grow business.

Architecture and User Interaction

For this particular customer, a particular innovation was one-click ordering and facilitation of interaction with partners. But can something like that be architected into business and development processes?

"Architecture is a key component of how you design how the user interacts with your systems," he says.

One-click ordering is a great example because it reduces the amount of clicks a user needs to do and enables customization of the user experience, making it easier for the customer to navigate the website and reducing latency of the interaction.

"That's one of the key things you should look for when you're architecting an infrastructure for user facing," he points out.

Five Steps to Network Nirvana

And speaking of architecture, Sinha also points out that a simpler architecture can be key in order to deliver a flexible architecture with consistent policies. Simpler makes governance easier and enables faster development.

The following are Sinha's recommendations for the key things to look for from a network perspective:

• How to virtualize service endpoints
• How to secure these endpoints
• How to deliver the scalable and highly reusable services
• How to offload (because reuse is the key goal and utilization and improving utilization is a key goal)
• How to accelerate and offload this processing because you want to reduce the latency for the user and then of course, management is a key goal as well
• How to create some policy driven IT infrastructure so that it’s easier to manage.

Executive Summary by Krissi Danielsson 

According to Lyndsay Wise, a BI analyst who just launched her own WiseAnalystics Web site, BI is really starting to make its mark in the mid-market.

"There are about three ways that market consolidations are changing the face of business intelligence deployments," Wise explains. The first is that consolidations are leading to embedded BI within vendors' service offerings -- or at least readily available adapters. Smaller and newer entrants are bringing best-of-breed solutions and giving organizations more choice. And last, but not least, the expansion of BI towards SaaS means that hosted offerings can be deployed at a departmental level.

Benefits and Pitfalls

Two key benefits of having numerous choices is obviously, well, there's more choice. Organizations that don't have large budgets or IT infrastructures can choose among the available solutions to find the most appropriate.

"Larger vendors such as Business Objects and Cognos are creating not only mid-market solutions but on-demand solutions as well, so that organizations in the mid-market arena that cannot afford or don't necessary want even a mid-market solution can still get the benefit of these leading vendors but through an on-demand service," she says.

As for challenges, the consolidations bring in more doubt in terms of which offerings will remain available in the next year.

"Organizations won't really know until next year whether or not the vendors that have been acquired will actually still have a role at their own development and their market positioning, etc," Wise says.

Newer entrants can also face viability questions from not having as much of a proven track record.

BI and SaaS / Open Source Development

Both SaaS and open source are becoming more mainstream, and BI has a definite parallel.

"A couple of years ago, it was really only open source organizations or companies or developers that were interested in open source that knew about open source BI solutions," she says. "But now we see that based on different marketing efforts and just expansions in general into the area, that open source vendors are actually becoming more mainstream within BI."

On-demand has the same story. What was once more of a fringe movement is now becoming far more mainstream.

"I think that both open source as well as SaaS-based solutions will continue to expand and become more popular within the BI arena," Wise predicts. "Especially in terms of mid-market solutions because so many mid-market companies don't have really advanced or mature IT infrastructures, it makes sense that they would go towards an on-demand solution."

Executive Summary by Krissi Danielsson 

Listen to Part 2 (18:00) Download file

Business can feel like a war sometimes, so it's no surprise that the ancient classic "Art of War" by Sun Tzu has relevance to the corporate world.

The Warfare Metaphor of Business

"Let’s look at what the business environment is about. It’s about competition," says Richard Platt, Intel's former corporate innovation program manager and senior instructor for innovation methods. "Warfare is really the ultimate competition."

Warfare, like business, has guidelines and rules and principles though -- and business can be Darwinian with survival of the fittest and most adaptable companies.

It's important in using this metaphor, however, to understand that business warfare is not against customers but about winning hearts and minds and gaining competitive advantage. It's about market share and influence in the mind of the customer.

Survival of the Fittest in Business

A researcher named Langdon Morris performed analysis of Fortune 500 and Forbes 100 companies over the past several decades and discovered some interesting trends. Morris found that between 1979 and 1983 that one third of the Fortune 500 went out of business or fell off the list and that the list had a 6% annual turnover. The Fortune 100 was almost as bad.

If you extrapolate the data, says Platt, and Project it forward, you find an interesting trend. Authors Richard Foster and Sarah Kaplan have predicted that only a third of today's major corporations will survive as businesses for the next 25 years; most will die or be bought out and absorbed.

"The reality is that we’re looking at drastically compressed planning horizons for every company and the need for fast response," he concludes. The root cause of the dropoff is the adaptability issue, just as the best fighters tend to survive a war.

Using Warfare Strategies in Business

So what does this all have to do with the Art of War? Sun Tzu said that the highest form of competing is to win without fighting. In business, that means winning customers' hearts and minds and to compete where competitors are not. One tool for accomplishing the latter is called the Blue Ocean Strategy of seeking calm waters, which contrasts with the Red Ocean strategy of sailing into bloody water.

Another tool, to resort to a military analogy, which be the OODA Loop, which was developed by an Air Force colonel. OODA stands for Observe, Orient, Decide, and Act.

"If we look at this from an agility standpoint, we’re really looking at culture, organizational culture, and a climate of trust that actually encourages people to actually use their initiative and further the goals of the organization," Platt says.

For more on the metaphor and for specific examples of techniques like the OODA Loop in action, be sure to listen to the entire podcast.

Executive Summary by Krissi Danielsson

Corporate innovation guru Richard Platt, formerly of Intel, has a lot of strong opinions about innovation. In two previous podcasts, he has described how Sun Tzu's Art of War applies to the corporate environment and why innovation is important for businesses to stay competitive.

But realizing the need to be innovative is only the first step. The second and most important is actually taking that step.

Challenges Companies Face in Acting on Innovation

The corporate structure itself tends to be a barrier to innovation at times, given that middle management is often expected to sort the good ideas from the bad before
ideas reach upper level management.

"Decisions made by mid-level management can be rather difficult and tenuous, because you get into political aspects and if, to any degree at all, a company is siloed, then you don’t get good sharing," Richard explains. "For the individual or the group that sort of comes up with a really neat idea and they’re starting off at the bottom of this pyramid of the corporate structure and sort of walking it through, I mean, you’ve got to be wearing track shoes for the bloody thing and from a corporate standpoint, his isn’t really a system or a pipeline for innovation."

Overcoming Innovation Barriers

So how can companies overcome these challenges? You have to put in place a process, Richard suggests, but the answer to that question lies in the definition of industrial or commercial innovation itself. Formulas and functions exist so that you can evaluate functions and costs with very specific formulas.

Industrial Innovators and What They Are Accomplishing

In researching innovation, Richard says that he spent a lot of time investigating who was actually using these strategies. One source of innovative companies is Business Week's list of the 25 most innovative companies. Numerous big-name companies like H P, Nokia, Siemens, and Samsung use innovation methods.

The vice president of Sigma Breakthrough Technologies developed a presentation evaluating development techniques across 233 manufacturing firms and looked at nine specific approaches. The data verified that when you listen to the customers and end users, companies are more speedy and profitable, and that having specific methods in place to accelerate time to market tended to pay off. A lack of a structure for innovation tends to mean reduced profitability.

"That’s what the data tells usm" Richard says. "It’s been very interesting to see data like this confirm what I had sort of worked out through my research and work in other areas that validates the very direction that Boyd is getting at is in the O-O-D-A Cycle of things: compress time and you’ll gain benefits."

Executive Summary by Krissi Danielsson

Bennett points toward the automotive industry to illustrate the point. Although the auto industry has its Big Three or Big Four just like the software infrastructure industry, individual automotive vendors target specific spaces, like the high-end or the low-end of the market. Some vendors try to cover every segment of the market and others target specific niches.

In the software industry, you see much of the same. Just like a vendor like BMW might specialize in creating a car to perform and handle better than other cars on the market rather than trying to create an option for every type of consumer, an integration technology vendor might focus in on a core business problem for a specific target audience and customize an offering to fit that market need, such as collaborative business problems like supply chain management, supply chain finance, or SOA compliance.

Specialization tends to evolve as companies trying to maintain a market share look for a way to sustain themselves in the face of competition. As companies look for a way to adapt in order to meet new business pressures, they need systems that can adapt to regulatory changes and new business models and changes in existing business models.

Axway's Approach to BPM and Business Agility

A lot of people pay attention to orchestration of BPM, said Bennett, but not everyone pays much attention to choreography. Axway's approach attempts to carve out a niche in the BPM field by being specialized to deal with both choreography and orchestration.

The solution attempts to help customers with business agility and enhanced collaboration as well. Axway aims to help its customers deal with SOAs and regulatory compliance in order to solve competitive challenges in the current market and allow companies to be more dynamic in response to market events, such as regulatory or sensor-related events.

"Having great domain in solving some of these SOA problems or collaborative business problems is key," said Bennett. "And knowing it from a contextual level, from a vertical perspective, understanding a healthcare/life science problem vs. a financial services problem."

Do Companies Need to Consolidate Their Strengths?

But with the trend toward specialization, is the market getting too fragmented? Are companies being forced to focus on their core strengths at the expense of innovation? 

Bennett doesn't think so. The need for agility and the need to compete will still drive companies to look for ways to distinguish themselves, but focus isn't necessarily a bad thing.

"I think you have to be focused. If you try to cover every market, you're going to fail. In fact, even the big guys, they have challenges covering every market," he said.

Case Studies

Bennett points out that Axway now has more than 8,200 customers, including most of the Fortune 500 companies. He cites DHL as one example, where DHL has developed a quick-shift product where about 20,000 trading partners and customers have automated key processes to create a collaborative backbone in order to extend new services to customers on demand.

McKesson is another example, said Bennett, of a customer that has built an e-commerce infrastructure on Axway's platform in search of agility. McKesson provides its customers with a way to distribute goods, as well as advanced services around ePedigree and controlled substance ordering systems. These services are provided over a backbone, taking the infrastructure and leveraging pre-built solutions in order to offer more benefits and values to customers.

The core technologies of ePedigree are an interesting example, according to Bennett, because the next step is track and trace which can work down to an RFID level or a sensor-based level in industries like automotive, aerospace, or supply chain event-oriented environments.

"As the next generation of supply chain moves towards supply chain finance, where you're tying together these sensor events with financial events, you're going to see new types of solutions for visibility, track and trace and supply chain finance that will drive revenue up for a lot of these companies and improve their bottom line," he said.

Finding a Win-Win Proposition

A lot of companies today in the infrastructure space focus on enterprises, but Bennett thinks a good strategy is to focus on collaborative business problems, because if you need agility in any area, that would be the one.

"If you have these collaborative business solutions in place and the right infrastructure in place, you're absolutely going to need agility," he said. "If you break anyone of those things, you break a lot. You can lose a lot of money quickly."

To find out much more about the topics summarized above, listen to the entire 15:07 podcast.

If the idea of carving up an SOA strategy sounds a little ghastly, ebizQ is scaring up some help this Halloween with the SOA in Action Virtual Conference. One of the keynotes for the conference, scheduled for October 31, is "Building a Strategic and Tactical Platform for SOA" with Forrester Research's Randy Heffner. Heffner joined ebizQ's Gian Trotta recently to offer a preview and some quick tips on SOA strategy.

SOA Misconceptions (or Tricks)

Web services and SOA are not the same thing, says Heffner, and viewing the two concepts that way tends to lead companies down the wrong path to a series of other pitfalls and tricks.

"Instead, view SOA as a broad set of design concepts centering on your major business processes and transactions,  and view Web services as one set of application-to-application communication protocols by which to access your services," he explains.

Another idea that follows that one is the idea that SOA is a technology only. That's a very small view of SOA, says Heffner, and what's more important is the idea of SOA as a business design to enable strategic transformation and flexibility to optimize processes, which is the higher level view of SOA.

And you don't necessarily need to buy loads of new products that break the bank in order to do SOA, he continues. "While it's true that you may eventually buy enterprise services buses and SOA management and repositories and appliances and such, you may well be able to get started on SOA and achieve strong business benefits without buying anything new."

SOA Truths (or Treats)

No one wants tricks, so how about some treats? Heffner says there are plenty of them in SOA. "The real treat within an SOA registry repository solution is not the registry, but the workflow and the service lifecycle tools around the repository," he states. That's what gets you a start on what makes SOA successful -- strong governance and organizational maturity to use a repository in the right way.

Considering SOA and Web services management, the real treat is visibility into the service implementation layer, says Heffner. A management product that looks only at requests and responses leaves you in the lurch if you need to troubleshoot something at the Java component or .Net component layer.

In an enterprise service bus, the treats may vary by the product but nearly all of them give you a unified access point for your services, he says, which gives you control over how services are accessed and how routing and various specifications are handled. And SOA testing tools give you strong and repeatable testing -- crucial as your company evolved and upgrades.

Heffner's Upcoming Keynote

Sound interesting? Heffner will be discussing all this and more in greater depth at his upcoming keynote.

"What we'll talk about in that presentation is focusing on the strategic view of your SOA platform," he says. "So, in some sense, starting with a more theoretical view in, 'Well, what are we aiming at? What do we want to get to with our strategic SOA platform?'"

A problem many companies encounter is how to get there while juggling different SOA projects on conflicting application platforms with different requirements and priorities for the infrastructure. And in order to put all that together, companies need a strong model for the SOA platform.

"You have to move towards that model by leveraging each project as a tactical step in a longer-term evolution towards that strategic platform," he says. "So in that session, I'll give the audience an overall view of what a strategic SOA platform is, what its major functions are, how some of the product categories of SOA infrastructure are playing out."

So in order to learn more about how to battle your company's SOA monsters, be sure to check back in to the SOA in Action Virtual Conference on October 31.

Executive Summary by Krissi Danielsson

Agenda and Resources 1. Office 2.0 Conference's iPhone Giveaway 2. What is Office 2.0? 3. How iPhones Relate to Office 2.0 4. Upcoming Product Launches Read a complete transcript of the podcast here Visit the Office 2.0 Conference website Learn more about SOA in Action Win an iPod -- Become ebizQ's Next SOA Idol! ebizQ’s second SOA in Action Virtual Conference give our entire audience a chance to win two highly desirable prizes – an Apple iPhone and the acclaim of our 115,000-strong community of integration practitioners. Submit your SOA solutions and innovations at the link below and watch the votes roll in as our members acclaim or flame your solution.

You have no doubt heard a lot about Web 2.0 and Enterprise 2.0, but what about Office 2.0? At IT Redux's upcoming Office 2.0 conference scheduled for September 5 through 7 in San Francisco, numerous experts and customers will gather to take a look at some of the mobile productivity and collaboration technology that may revolutionize day-to-day life in offices around the globe. And in the spirit of Office 2.0, they're going paperless and giving away an Apple iPhone to each attendee for use during the conference.

"We're promoting the idea of doing everything online, and there is no better way of proving that it works by actually using online tools for managing the company itself," says event producer Ismael Ghalimi about the iPhone giveaway.

But why are iPhones symbolic of the spirit of Office 2.0? Ghalimi has more to say about that.

The Idea of Office 2.0

Office 2.0 is the extension of Web 2.0 into office productivity tools. It's the idea of getting office-type work done from anywhere using online technology and services through a Web browser rather than solely through desktop or laptops in the office.

"It's doing everything online, putting everything in the clouds," Ghalimi explains. "The primary reason for that is, when your data is online it is much easier to share with people, and more and more your work gets done with your peers, with your colleagues, with your customers and partners... When you can share information online it can make you more productive, it's going get things done a lot faster and a lot easier for a lot less money."

So, Office 2.0 collaboration tools might be anything from word processors to spreadsheets to database presentations and collaboration tools all delivered through a new model.

What iPhones Have to Do With Office 2.0

According to Ghalimi, iPhones are the perfect vehicle from which to demonstrate the ability of online tools and services to get the job done. At the Office 2.0 conference, attendees will be given iPhones and then instead of a printed agenda or schedule, they'll be able to connect to the Internet and go to a website with the information. They will also be able to connect to a directory to get contact information of other attendees with whom they want to network, or even to get a live video feed of sessions.

"I think we're going to demonstrate that what you get with that device is a real-time tool that makes it a lot easier to collaborate with people when you're on the go," he says.

Attendees will also be able to try out a lot of really neat applications through their iPhones, says Ghalimi. And of course there's a backup plan for those that already have an iPhone.

"For people who already have an iPhone we're giving a Sony Playstation 3 on which we've installed Linux, and so we can run the Firefox Web browser, which plugs into your TV and now you can not only play games but you can also do word processing from your sofa," he says.

New Office 2.0 Product Launches

So what else is the big buzz for the Office 2.0 Conference? Well, last year Google announced Google Docs and Spreadsheets at the Office 2.0 Conference. Expect even more this year, says Ghalimi, in the Launch Pad even where 20 new companies will announce new products and services on Thursday September 6th during lunch break.

"I can't tell you what this is all about; you're going to all discover that at noon on September the 6, but there are some very very interesting applications that will be released -- especially several of them directly working on the iPhone so we're very excited about that," he says.

Executive Summary by Krissi Danielsson

A recent ebizQ Webinar guest estimated that by the year 2010, the world's data volume will reach one zetabyte -- a 1 with 21 zeroes after it. To state the obvious, that's a lot of data, and it's also an obvious challenge for corporations  that want to find ways to use that data to business advantage. Bath, U.K.-based CopperEye is one company that hopes to help solve that problem.

Complications and Costs of Data Management

The first challenge of data management is obviously being able to find the data you want, points out CopperEye CEO Kate Mitchell.

"As companies have been continuing to increase the size of the database they are adding larger hardware platforms so that performance remains constant, both on the transaction side and the analytical side," she says.

But databases are growing so large that hardware costs and data center footprints, along with power requirements, are becoming unmanageable and companies are starting to have trouble finding highly skilled DBAs to manage everything.

In some industries, traditional solutions can still keep pace with data needs, but in others you cannot just throw money at the problem, Mitchell says. The explosion in data is too dramatic. And furthermore, she points out, regulatory issues are forcing companies to keep more data for longer and be able to provide specific responses to inquiries sometimes in 24 hours or less.

Data Management in Industry Verticals

The telecommunications sector tends to have complex data needs, says Mitchell. "They are heavily regulated and in this case, aiding to combat terrorism - they need to find very specific information on individuals that are potential criminal suspects -- to be able to find calls they made, when they made them, where they were, who they called, how often they called."

In business, issues like 3G are creating loads of data, as are services like cell phones for audio/video shopping and so forth. The financial industry tends to have particularly intensive data needs as well. And as websites are being asked to track visitor information, that creates massive amounts of data as well.

Scaling with the Data Explosion

Relational databases have weaknesses in managing huge amounts of data, says Mitchell. Relational databases were designed to manage data that's changing. "There's no better approach for concurrency with hundreds or even thousands of users inside an organization and outside an organization and making sure you've got all of the capability for transactional integrity whether that's two-phase commit or row-level locking," she says. "That's exactly what the database was designed for."

But in data that's not changing, such as transaction or event data, sometimes you may not need the overhead or complexity of a relational database. An alternative is to keep such static data in a lower cost, scalable location such as a simple flat file -- but without giving up immediate and precise access to that data.

"That's what CopperEye is finding that our customers are pleased with as a new innovation in managing this vast volume of data without giving up very precise and immediate access to the data," she points out.

CopperEye takes the flat file system and tries to use the best attributes of that -- the low cost and high scalability along with simplicity and indexing, almost cherry picking through the IT stack.

Case Studies - A Wireless Provider and a Messaging Company

One CopperEye customer, Orange (a UK wireless provider) needed to track anomalous calling patterns for revenue tracking and for regulatory compliance with the EU. Orange found that storing all that data was not feasible in a relational database model, so they worked with CopperEye to find a way to store the data in a flat file instead.

"We added up the other day that we've handled 500 billion transactions for them over that period of time [seven years]," Mitchell said.

Orange went from storing call records in a relational database, where it could only afford to keep ten percent for two days, to keeping the records in a flat file where it could keep 100 percent for 40 days. "Just to put that in perspective, to meet the new EU mandates, rather than keeping all this data for 40 days, the guideline is to keep it for a minimum of a year and some EU member countries, like Ireland are saying that data needs to be kept for three years," said Mitchell.

In another example, a CopperEye customer called Message Labs had over 13,000 corporate customers with 35 million email accounts and about 8 billion emails per month. The company needed tot rack these emails across twelve data centers around the world with a 24-hour service level agreement that Message Labs would locate missing emails within 24 hours.

"Their network help desk would notify their operations people," she said. "Their operations people would actually be trolling through these log files trying to find literally that needle in the haystack; that one email out of the billions that they handle in any particular month."

And customers wanted turnaround faster than 24 hours a lot of the time. Message Labs implemented CopperEye and got direct access to log files. Now, within a few seconds, customers can go online and enter recipient or sends or subjects and retrieve information on a particular email within a few seconds. This has improved customer service and lowered costs for Message Labs.

Flat Files and Vocabulary Search

Mitchell is quick to point out that flat files are not a cure-all for all data storage needs. "This approach is not for text or word documents or web pages," she says. "CopperEye is focused on data that would otherwise live in the database -- scalar data."

For e-discovery and searches for documents and Web pages, companies like Endeca, Fast, Google, and Yahoo tend to be most optimized. "They pre-build all the indexes based on every important word or relevant word