Enterprise Mashups in Action

Is Your IT Organization Ready for the Attack of the Angry Birds?

2011-07-11T20:44:54Z

It sounds like the setup for a joke: 'What happens when you bring Angry Birds into your Enterprise?' But the punchline is anything but funny: 'Nothing until they make a nest in your data center and start eating your data.'

In Act 1, the business world went Enterprise App crazy. Your IT department created an App for this, and an App for that. Before you knew it, you had more than a webpage full of them. Now take a minute and think that process through to its logical conclusion. You can see where this leads, can't you? Act 2 is Enterprise App Stores.

Enterprises have always pursued simple and effective ways to secure, govern and distribute content to their employees, customers and partners. That's why corporate portals became so popular; organizations saw it as a single place to put everything. In fact, I'd say that enterprises are beginning to see Enterprise App Stores as their next-generation enterprise portal.

But there a few new twists in this story and, like all new technologies, the buyer needs to beware. So the minute your CIO says 'I want something like Apple's App Store in my organization and I want it now!', remember these words: 'it's all about security, stupid' (or something less career-limiting, if you want). And if you really want to dumb it down, tell your CIO that you first need to address the 'Angry Birds Problem'.

You see, in the Angry Birds Enterprise Edition, your birds are accessing your enterprise systems. Consumer Apps (like the original Angry Birds, as well as Angry Birds Seasonal Editions, Angry Birds Rio, and even the rare Angry Birds Magic) don't access things like your payroll system, your personnel records, and your accounts receivable. However, you, as the lucky creator and distributor of Angry Birds 'Enterprise Edition', have to solve this puzzle.

Let me put it CIO-speak again. If we published Angry Birds Enterprise Edition, we'd need to know what kind of birds are they (what are the App features)? Where did the birds come from (what datasets do they access)? Why are the birds angry (what security does this App require)? And, last but not least, will our bird house (our Enterprise's App Store) have what it takes to support all of our App's data access and data security requirements?

Apps inside the enterprise cannot get away with lax security and governance. There's authentication, authorization, and firewalls to worry about. There's access to enterprise systems such as SAP, PeopleSoft, MicroStrategy, and SharePoint, as well as all your trusty home-grown applications. And since you have to solve these riddles in a way that SCALES (i.e. not just for one App, but for hundreds), your distribution tool, i.e. your Enterprise App Store, has to distribute your Apps without breaking any of your security and governance rules. It's an easy requirement to overlook but a critical one.

The point here is you need to look closely at anyone claiming to have a "Enterprise App Store solution". Much like the tools you use to create your Enterprise Apps, tools to distribute your Enterprise Apps have to handle some very enterprise-y requirements. You should at least ask the following questions of your Enterprise App Store platform vendor:

1. Do you plug into my identity management systems?
2. Are you able to provide authorization policies around my Apps and data sources?
3. Do you provide end-to-end security from my App all the way back to the datasources?

If the answer is "yes", then you've at least covered the basics. However, if any of these is a "no" you may want to look elsewhere for that enterprise-grade birdhouse.

Guest Post: Getting a Chokehold on Your Data

2011-05-11T21:07:57Z

A quick note: I often hear great ideas and see great blogs from peers and co-workers. Here's a great writeup from Americo Savinon, a genuine Mashup expert and an aficionado of the martial arts.

-------------------

I'm pretty sure you've heard the term 'Mashup' before. Maybe on TV shows like Glee, or radio shows like 'The Morning Mashup'. So I expect you understand the idea of mixing two or more things together, like concepts, ideas or songs, to create a new entity.

Nowadays, Mashups are a common term we often hear not only in the entertainment industry but in the technology world. As a Mashup Engineer, I often deliver a data Mashup from many disparate information sources and hear people say 'I've never seen my information combined (aka mashed) in this way'. The opportunities awarded to us by our ability to 'mash up' information is why I believe Mashups have changed the way we think about solving business problems.

The central theme of Judo is to "deliver maximum efficiency with minimum effort." Translation: your opponent pushes, you pull; your opponent pulls, you push. This applies to the world of Mashups because the premise is to create new insight by "leveraging and adapting", not "changing and recreating". Jason Fried, the co-founder of 37Signals relates Judo to software using the term "Judo Solutions". He says; "a Judo solution is a solution that delivers maximum efficiency with minimum effort". That's a mashup! New information that was not created from scratch, but rather made from mashing existing information together. Low effort, very efficient.

According to judoinfo.com, "Judo which is translated as the 'gentle way', teaches the principle of flexibility in the application of the technique; skill, technique and timing, rather than the use of brute strength, are the essential ingredients for success in Judo". In other words, as Jason put it," When good enough gets the job done, go for it." I am practicing this very principle with every Mashup I make. And no, this does not translate to 'laziness'. Remember: efficiency.

So how can you start using Mashups to create your own Judo solutions? I follow a series of steps I call "The Mashup Solution":

Define the Answer to the Problem: Take the 'good enough' approach and follow the 80% rule. Ask yourself, 'What gets the job done'? Be sure to understand the fundamental problem first, and don't worry too much about the details.
Visualize the Solution: Create an image in your mind of how the final result should look/feel/sound.
Do it Fast!: Think minutes, hours and days - not weeks, months and years. Businesses need answers quickly.
Iterate the Solution: Get to work and don't hesitate. Trust your choice of technique and tools. Then iterate in short sprints to get it right.
Share and Refine: Get the solution out there as soon as possible. Get others to try, test, and critique it. Be open to feedback, criticism, and praise.

In applying mashups to enterprise information needs, I've found that the 'perfect solution' almost always follows in the footsteps of a Judo master: maximum efficiency and minimum effort.

-------------------

The New BI is Just-In-Time

2011-04-01T18:26:24Z

Earlier this year I had the pleasure of visiting the Boulder BI Brain Trust (BBBT) where I had the opportunity to sit down with a great group of well-known BI experts. These aren't just "BI folks" but some of the founding fathers and mothers of Business Intelligence.

As expected, we spent a lot of time discussing BI in general and specifics such as data quality and self-service. But our attendance wasn't to talk about traditional BI. We were there to get their thoughts on a new BI niche we called "Real-Time Intelligence."

To help facilitate a common understanding of what we meant by "Real-Time," we presented a chart (see picture below) that describes the problems which best fit real-time solutions. I think it conveys an important message: the right context for real-time intelligence solutions are situational problems with hard-to-predict, variable data needs. That's certainly not your typical use-case for traditional BI.

Boris Evelson at Forrester might describe this style of BI as 'Agile'. In his just-issued a report, 'Trends 2011 and Beyond: Business Intelligence,' he describes the future of Business Intelligence as agile, and describes some of these same characteristics.

We've found that implementing a real-time BI system can lead to a number of new, untapped organizational improvements. Claudia Imhoff (one of our many BBBT friends) states in a SearchBusinessAnalytics.com article entitled "Getting Started on Building a Real-Time BI System: Issues to Consider," that real-time BI can lead to "faster decisions based on more timely data; better customer satisfaction due to improved customer service; reduced "opportunity costs" in marketing and sales; and supply chain management savings through just-in-time deliveries and reduced inventory levels."

In that same article, Colin White (another BBBT-er) describes the "analyze-and-store" model, where users analyze data as it is run through a system to be stored. The data is real-time, the delivery is real-time, the results are real-time.

And I think the Real-Time style of Intelligence compliments the more traditional, historic BI approach very well. Imagine being able to get up-to-the-minute operational information from a wide range of systems, mashed with a deep historical record, all displayed in a context-based and visually-rich dashboard. That combination of past and present can have a powerful impact in decision-making.

This small trend to move away from a 100% reliance on the data-warehouse-driven "store and analyze later" approach has been recognized by other industry journalists as well. Rick van der Lans, in his article "JackBe Presto: A Self-Service, On-Demand Data Integration, Mashup-Based, Dashboard-Oriented, Business Intelligence Tool", discusses some of the benefits of a on-demand intelligence solution.

As you can see, Real-Time Intelligence has gotten a lot of buzz in the press. If you're as intrigued as we are, then join in on the discussion and tell us: Do you think Real-Time Intelligence is the "New BI?"

2011 Predictions: It's Really Time for Real-Time

2011-01-11T18:28:22Z

This year I decided to do things a bit different than past years. Before placing my first bet on this year's prediction list, I decided to wait and listen to what 'the experts' were saying about 2011 technology trends. I expected iRobots and Google's first moon flight, but no one seemed to raise the ante. What I heard was a lot of the same; cloud, BI, mobile, tablets, virtualization and more cloud. I hate to downplay their predictions but most of what I saw was just a recap of IT topics-of-the-moment. At the same time, I reviewed my predictions from last year and there 'it' was, lucky number 7:

"2010 becomes the 'Realtime' Year. Enterprises want their information as fast and as realtime as the Web. BI vendors struggle to get to anything close to realtime and begin telling customers they need to begin looking at other technologies for realtime data, such as Enterprise Mashups."

I may have been a bit too early and a bit too specific, but I am confident this is one topic whose time has come. We continue to go to work and use 'warehouses' of information that are stale, disconnected copies of our actual live, dynamic data. That's an old design from the 1990's and one we no longer have to live with.

I believe in lucky number 7 so much that I am making it the focus of ALL of my predictions for 2011. I am calling this the "Real-Time Edge": getting real-time information to the user by leveraging the computing power at the edge (tablets, mobile, sensors, etc).

So, here it goes, my top ten 2011 predictions in real-time. (Trust me, you haven't heard any of these before):

1. "iClouds" join Apple's "iFamily." Apple announces the "iCloud" specifically designed for iPad real-time intelligence Apps.

2. Gartner and Forrester face off in lengthly legal battle. Gartner and Forrester both formally announce a new real-time BI niche called Real-time Intelligence (RI). Unfortunately, both companies have a very public legal battle over whether 'real-time' actually contains a hyphen.

3. BI lacks tradition. IDC puts out the first "Traditional BI Vendors doing Real-time Intelligence" point paper which interestingly enough doesn't list any traditional BI vendors.

4. Amazon ditches the Kindle for Robots. Amazon and Willow Grove announce the world's first real-time OpenCV (Computer Vision) cloud processing engines that can be used with personal iPhones, Android, camera-based tablets and even robots.

5. Oracle attempts Real-Time Real-Time. Larry Ellison mandates that all Oracle software products include "Real-time" in their name. This includes renaming the only product that already has "Real-time" in its name: "Oracle Real-Time, Real-Time Decision Server." Sounds weird.

6. Google renames Groupon. On April 1, 2011, Google announces a $10B bid for the new real-time GroupOn who just renamed themselves "RealOn."

7. SAP thinks they're sly, in Real-Time. SAP insists all their products have been real-time since day one. No one believes them.

8. Microsoft bets on "Real-Time Steve." Microsoft announces "Real-time SharePoint 2011" and touts that that every single SharePoint instance runs in memory, letting anyone find any data in any SharePoint instance within 2.4 milliseconds. Steve Ballmer renames himself "Real-Time Steve".

9. The Cloud Confusion continues. Everyone else announces a real-time cloud offering and calls it the "Real Cloud". Unfortunately, no one really knows what it is. Sorta like today's Cloud.

10. I'm suffering from a mid-life crisis. I get a "Get Real-Time" tattoo.

I wanted to add more but my calendar says I have a real-time meeting with a real-time customer. Happy Real-Time New Year!

Are You in the BI Abyss?

2010-12-03T14:37:00Z

A few months back I attended a 3-day conference on Business Intelligence, hosted by one of the big analyst firms. It took me less than 15 minutes (listening to the keynote speaker, in fact) to realize that the general mood wasn't good. In spite of ~15 years of history and a recent surge in interest and functionality, its acolytes and adherents seemed to be haunted by a litany of issues.

After leaving the conference I realized this mood shouldn't have been a surprise to me. I had heard a lot of criticisms from my customers and partners in the last few years about 'BI gone wild'. If I had to turn the issues into a parable, I think the story would sound like this:

Acme Corporation just bought a brand spankin' new BI system for the business units to have sales, inventory and manufacturing dashboards and reports. It is intended to replace a time- and labor-intensive process involving lots of spreadsheets and manual effort. A few years later the reports and dashboards start rolling out. Everyone hates something. The reports are too ____ for sales. The dashboards are too ____ for marketing. The exec team says none of the data matches their needs. In the end, everyone asks to have the data dumped back into spreadhsets so they can do they're own number-crunching and reporting. You're back to where you started, with a slow, labor-intensive system.

Sound familiar? And this kind of requirements-results mismatch creates an 'Abyss', the gap between the information needs and the solution your organization would use to provide them.

In a recent post by Peter Schooff from eBizQ, he asked 'What's the Next Big Thing in BI?' I believe we are now at a time where we are seeing a BI Abyss emerging, due to factors like:

1. Business users want more control. They want to be hands-on, instead of letting the BI blackbelts choose when, what and how they see their information.

2. Business users want live data. This is the only way to respond to the dynamic world they now operate in, not historic 'rear view mirror' data.

3. Business users want adaptability. If they need a new data feed added to their dashboard, or a new computation added to a report, they're gonna expect it to be immediate.

As more tech-savvy users hit the workforce every year, I think this trend will only increase. Ultimately, I think we see an entirely new breed of BI, a kind of 'Real-Time Intelligence', that will be the de facto approach. Old school BI, with long-cycle ETL, warehousing, and report-development, will be relegated to a niche of computation-intensive decision-support problems.

And not all of the BI vendors are oblivious to this opportunity. Products like Qliktek Qlikview and Microsoft PowerPivot are good examples of new technologies that can help cross the BI Abyss. They are BI with more control, more liveliness, and more adapatability.

So ask yourself: are you in BI Abyss?

You Can't Stop (or Fight) the Pendulum

2010-11-09T18:57:07Z

BusinessWeek Online just published an interesting article called 'Mobile Business Apps Flourish at IBM, Google.' As a constant media-watcher, I can tell you that it is quite a milestone when the mainstream business media starts to bang the drums on something like this. And I think this was inevitable.

I call this the 'TenPen' Trend. Just like Moore's law where 'the number of transistors in microprocessors doubles every two years,' there's a similar trend in software. TenPen states that approximately every ten years (software moves much slower than hardware) the momentum and focus of the industry swings to the other side of the market.

Here's the theory: Unlike hardware, which gets smaller, software gets bigger, but only on one 'side.' Software of course, has two sides, which I imaginatively call 'the backend' and 'the front-end.' (catchy, isn't it?) The backend is where all the heavy processing takes place. It's the middleware, app servers, ESBs, portals, etc. The front-end is what the user sees and it's usually very user-interface centric.

But the TenPen Trend states the software industry only has energy to focus on either the front-end or the backend during a period of time. So, approximately every ten years the focus swings to the other side allowing us to innovate and grow that side in size for about ten years. Near the end of the cycle, that side begins to overshoot the market in size and complexity and begins to commoditize. Then the innovation shifts to the other side.

Case in point: as Java Enterprise Edition (JavaEE) grew larger and larger, overshooting many, a group of developers were building Java based frameworks that were much lighter and agile than JavaEE. The success story is SpringSource, which developed Spring. SpringSource was acquired by VMWare in 2009. Approximately ten years after it started to get rolling.

So Apps were inevitable, I think. From 1997ish to 2007ish much of innovation and investment was in Java/.Net and back end systems like portals, app servers and integration. Around 2007 the industry started moving towards the front-end, starting with Ajax, Flash and then richer things like Silverlight. Then toward Google's Chrome browser, Apple's iPhone, the iPad and now Apps.

And, as we shift to this front-end of the pendulum, we also shift some of the processing, doing things that were traditionally done on the backend. That's why we're beginning to see 'portal-less portals' that do most of the work in the Browser. Mobile Apps that do a significant amount of processing on the device. And now that the business folks have gotten wind of Apps, all that remains is for this swing of the pendulum to fulfill its promise to provide the next big productivity boost for the enterprise.

Apps are the next serious business. Party like it's 1999. Again.

Should Enterprise Mashups Be Part of Your BI Strategy?

2010-05-10T13:34:55Z

If you're not a BI expert (like me) and you went to Gartner's recent BI Summit in Las Vegas to feel good about BI, you would have been terribly disappointed. This had nothing to do with Gartner, they put on a great conference. It was more about the state of affairs of the BI market. "We have to align the Business with IT for better results." I wish I had a $10 chip for every time I heard this said by analysts and customers alike. Seriously? I would have sworn that this is now common sense in the IT world, not some new age enlightenment: If you are building for the business, make sure the business is intimately involved. We've been aligning IT with the Business for the last decade, haven't we?

I'm being flippant and serious at the same time. I do believe IT wants to give the business what they want, but the problem is the nature of how BI systems must be built. Sure, the marketing is sexy: 'Instant Insight into your Business. Realtime Enterprises need Realtime Data. Optimizing your Value Chain with Analytics.' But the devil is in the details. The amount of work to get the data, cleanse the data, normalize the data, and build efficient queries on the data can be massive and take years. One customer at the conference (of one of the 'big 4' BI vendors) said it best: "We spent four years building out our BI system. We ended up with 1,000 BI reports and zero business intelligence." Four years to find out you failed!

I'll go out on a limb and propose that the BI problem is much the same as the traditional 'big bang' ERP implementations: large software systems require so much work to fulfill the core requirements that there is little time left to incorporate flexibility and agility. Sure, some in the BI industry have made efforts to address their problems. For example, some are trying to reinvent the way BI does ETL, data warehouses and data marts. Or to empower business users to help solve the problem "self-service" style through solutions like Microsoft's upcoming PowerPivot Excel plugin.

But I got really excited by Gartner Analyst Kurt Schlegel when he cited "data mashups" as one of the nine BI technologies to watch. I couldn't agree more. I had the chance to listen to Seth Grimes, the deep-thinking BI pundit at Intelligent Enterprise, talk about the natural synergy of Enterprise Mashups and BI. He called it 'Nimble Business Intelligence' and described it as 'flexibility, self-service, speed-to-insight.' Seth concluded with perhaps his most important point: Enterprise Mashups do not replace BI solutions. They sit side-by-side, complimenting each other. Certainly, Enterprise Mashups can consume BI services and make them part of mashup apps. And the opposite is also true.

BI and Mashups? I admit I am no BI expert but there seems to be a lot of potential for synergy between the two technologies, I think. I see this as the perfect way for the BI industry to get closer to the business user and farther away from the four-year-BI-doesn't-meet-our-user's-needs debacles. Mash on!

My Company Went to the Cloud and All I Got Was This Infinitely-Scalable, Highly-Available, Community-Driven, Collaborative Enterprise Mashup Platform in 30 Days!

2010-03-24T15:01:25Z

Cloud, Cloud, Cloud. Everywhere you look, someone is talking about "The Cloud." Check out the Google Trends chart below. I'm pretty sure Cloud computing is not a fad, but if you're like me, you're: excited, confused and a bit skeptical.

The news volume for 'SaaS' (think SalesForce.com) has had a nice slow ascent since 2004. But the trend line for 'Cloud Computing' is explosive, beginning in late 2007 (around the time Amazon's EC2 Cloud graduated from Beta). 'Cloud Computing' starts with a rapid take off and keeps going. With a rapid ascent like this, it's either a short lived fad or a long-term trend that starts with a lot of hype.

A quick tour through history can help you understand how we got here. In 2005 (yes, only five short years ago), Sun launched a brilliant marketing campaign around 'Utility Computing' which promoted a $1-per-hour-per-CPU pay-per-use model. (You can listen to Jonathan Schwartz, Sun's CEO at the time, talk about it.)

Unfortunately, Sun's target wasn't just any enterprise who wanted to run their software on the Cloud, but rather big financial services companies that required lots of computing power for a limited amount of time for things like Monte Carlo simulations. That's probably why they called it a 'Grid' and not a 'Cloud.'

Soon after, the big bad Internet retailer Amazon launched a preview of their "Elastic Cloud," or EC2 and Amazon had a very different model from Sun. They opened up their infrastructure to anyone who wanted to run their software on it. The catch was you had to do all the setup yourself and all you really had to work with was a Linux AMI on a small/medium/large box. There wasn't even a persistent file system at the time, so you had to do a lot of work to run a web application that actually saved data.

But even with these limitations there was something very liberating about getting an EC2 account and starting an instance with whatever version of Linux you wanted, all for a small per-hour fee. You could shut down your instances at any time and pay nothing until you started them up again.

Within the next few years Amazon added all the bells and whistles: lots of AMIs, lots of machine configurations, persistent file systems, a load balancer that manages health, geographic separation, SLAs and even a database as a service. It was this simple cost model in combination with a sophisticated platform that motivated my company to put our enterprise mashup platform, Presto, on EC2 last month. We call it the Presto Cloud Community Edition and all 4,000 of our mashup developer community members use it.

I love telling the story about how easy it was to get running. It took less than 30 minutes to get an instance of our software platform running on the Cloud. Yes, it took a bit longer to 'teach' EC2 how to get it running in high-availability (HA) mode. But now that we have that working as well, we simply save out the AMIs and just start them when we need them. Instant cloud-based HA!

Once you realize that YOU can have an infinitely scalable, highly available, collaborative platform for a fraction of the price it would cost you in your own datacenter, you start to realize the power of the Cloud. Maybe the hype is actually worth listening to this time.

With Enterprise Mashups, Sometimes 1+1+1=I've Never Seen THAT Before

2010-02-22T18:50:51Z

We all know the cliché: the result is greater than the sum of its parts. Quite often that's how Enterprise Mashups are described, 'combining things to produce something more,' a 'better version of the truth,' a more complete view, or simply 'connecting the dots.' But these are ways of seeing what you have in a better way. There is another, often overlooked, function of mashups as well: Enterprise Mashups as a way to create something new and different. And I think it is this aspect of mashups that truly separates them apart from some of their software brethren. But before I explain, let me digress.

With music mashups you mash two or more songs together. Listening to each song separately, you may have a good idea as to how the mashed song will sound. Here's a simple example: DJ Dangermouse mashes the Beatles White Album (circa 1968) with JayZ's Black Album (circa 2003) to create the Grey Album (circa 2004). But sometimes the song mashup has a completely new sound. The result isn't greater than the sum of the parts...it's something new entirely. My favorite example here is 'Bounce That' from Gregg Michael Gillis (AKA Girl Talk). You get major mash points if you can name all 11 songs in this one.

Similarly, Enterprise Mashups can provide completely new enterprise data insights. We're not talking about looking at existing data in a new chart or with another drilldown, like what BI technologies might help you do. And here's the real reason behind this phenomenon: we're talking about gaining completely new insights by connecting new data with new data in ways never done before. I often hear users say "I've never seen that data." Now, they're not saying that they've never seen the individual data sources before. But rather, they're saying they've never seen the results from those sources connected in a meaningful way.

Here is an example of NEW. Datacenter Support engineers have to make support decisions based on data from many different systems throughout the datacenter. This might include application log files, SNMP traps, fault management systems like NetCool, trouble ticket systems like Remedy and topology services. Now put it all together with a geospatial overlay of attack origin locations for a 'Cyber Defense' Mashup. Attacks stemming from a specific location or over a limited time period can be now (easily!) correlated with other attacks based on similar characteristics. Previously unrelated attacks could be shown to be part of a distributed denial of service attack. That is NEW.

I guess this isn't much of a surprise when you think about what Enterprise Mashups do best, i.e. combine and remix disparate and un-connected data (with emphasis on 'disparate' and 'un-connected'). Enterprise Mashups connect what might have been technically impossible or time-/cost-prohibitive in the past. That is the sweet spot that hasn't been addressed by other technologies.

Connecting siloed-and-sometimes-never-been-connected-before un-connected data sources can bring an analyst or executive out of their chairs. In one recent demo I mashed a little BI data with some other internal and external data sources in just a few minutes and could have sworn the BI IT team was near tears. It wasn't simply more, it was new.

And I think there can be even more to the story. So far we've talked about mashing real-time data and this provides a sort of 'virtual snapshot.' I say virtual because if I run the mashup again, I'll get the latest (and newest) data. The refresh button on that mashup may actually show you a new result every time you press it. This is great unless you also want to compare these results to those from 5 minutes ago.

Luckily, Enterprise Mashup vendors (like me, I admit) are beginning to understand that it can be very useful when an analyst can "freeze" a mashup and share the point-in-time mashup via Widgets and Dashboards with others. And we should then be able to mash these 'snapshots' into trends (we call it 'real-time trending') and then mash those trends with other disparate data to add yet another temporal dimension to the mix. Analyzing historical trends has always been the job of the BI OLAP and ETL vendors, but snapshots and trending will soon became a competency of the more sophisticated enterprise mashup platforms as well.

So, the next time you see a mashup, consider whether it is simply better than the static report or spreadsheet it replaced or if it is entirely new.

The Information Sound Bite of 2010: "We Failed to Connect the Dots."

2010-01-22T17:40:29Z

After the attempt on Christmas Day to destroy an airplane on its way from Yemen to Detroit, President Obama said; "There was a mix of human and systemic failures that contributed to this potential catastrophic breach of security. We had the intelligence information, but failed to connect-the-dots." Even the Declassified Review into Flight 253 says; "the problem was a failure to 'connect the dots' rather than a lack of information sharing." This "connect-the-dots" statement resonates well with me since "connecting-the-dots" is exactly what Enterprise Mashups do. To understand this, let's dig into the facts a bit.

The 23-year-old Nigerian who attempted the attack, Umar Farouk Abdulmutallab, was in TIDE, the Terrorist Identities Datamart Environment owned by the National Counterterrorism Center in McLean, VA. The database stores information about hundreds of thousands of individuals who may be a threat to the United States. The FBI Terror Screening Unit goes through the TIDE database and identifies those who are the real threat and places them in their Terrorism Screening Data Base, which contains about 400,000 individuals. Then the FBI passes these names to TSA, who maintains its own "no-fly" list of about 14,000 people. Because the FBI did not deem Abdulmutallab a real threat, he never made it to TSA's "no-fly" list.

There's a few ways to look at this problem. One way is to say the FBI failed in their analysis by not putting Abdulmutallab on the list. But, I believe the real problem is how the data was shared or, in this case, not shared. This common practice of copying data and sending it to others and calling it 'sharing' is really problematic for many reasons:

• Disconnected Data: Getting a copy of data means the data is disconnected from the original source and more importantly may not contain all the information needed. So it is 'stale' and possibly incomplete. Seems like something as important as Terrorist watch lists should be real-time, not 'copy and send.'

In the Government, there is a notion of trusted data coming from a 'trusted source', also called the 'authoritative source.' Not only do you trust the source, but you trust you're looking at the exact same data they are. When you copy data you have trusted data, but no authority other than yourself. The fact that the data has been copied now puts the burden on the recipient to ensure it has not been tampered with.

• False Sense of Context: Of course TSA analysts want to know why someone got on the "no fly" list. There would be no way for TSA to put their context around an individual who purchased a ticket in cash and has no luggage with the fact that his father reported him to the US Embassy in Nigeria. Context really makes a difference here. Consider this statement from the official government assessment: "A failure of intelligence analysis, whereby the CT community failed before December 25 to identify, correlate and fuse into a coherent story all the discrete pieces of intelligence held by the US Government related to an emerging terrorist plot again the US..." Substitute "coherent story" with "context" and you'll see what I mean.

So how do Enterprise Mashups help fix a problem like this? EMPs stop the copying/sharing and make the authoritative source available with the proper security and governance. A good Enterprise Mashup Platform (EMP) can ensure that only the users with the proper credentials gain access to the full or limited set of information. EMP's can provide this sort of "secure veneer" that connects directly to the same source, but constrains, or limits the data based on the user's credentials and policies. If this was done to the TIDE database, TSA analysts would have been able to see Abdulmutallab's information and "connect-the-dots" with real-time information such as all cash ticket purchase and no baggage.

When data is everywhere and the fusing and correlating must be done by a knowledge worker (aka analyst) you need to give the analyst the tools and time to analyze the data. EMP's are built to support real-time "connect-the-dots" analysis and are especially good at connecting the dots when the dots are all over the place.

EMP's dramatically reduce the time and effort required to gather the data and connect it together and they can do it in a way that is as safe as any person (or government) would require. Maybe the next time President Obama will replace "connect-the-dots" with "mashing the dots."

10 for 2010: I Predict Google Gars, Apple Books, a CNN Moment and More

2010-01-11T13:58:27Z

It's that time (again) when 'arm-chair visionaries' like me sit back and attempt to predict the next 12 months of technologies. In preparation, I did a quick search to see what others predicted for 2009. Some were better than others. The more conservative prognosticators predicted a Twitter growth explosion and growth in interest in Cloud Computing. Edgier predictions included 'email will die' and 'WiFi will be ubiquitous'. And of course there were some things no one predicted, such as Amazon digital book sales would outpace paper book sales on Christmas Day 2009. But overall I think the 2009 predictions were pretty conservative and unexciting, perhaps due to the looming recession.

However, as the US economy is starting to stabilize and we have a year of pent up technology cabin fever, I predict 2010 will be a technology innovation 'hockey stick year' (that visionary-speak for exponential growth). So, with that high standard in mind, here's what my crystal ball tells me:

1. Apple brings sexy back to books. Apple releases the 'LiveBook'. It looks like a 7" iPhone but it's meant for traditional book reading and the newly created Apple iPub publishing platform. LiveBook not only let's you interact live with books using iPhone touch gestures, but let's you interact with LiveBooks using text, live video, interactive widgets, social networking and collaboration. You can also write you own LiveBooks and publish via my iPhone Apps.

2. Enterprise Mashups have a 'CNN moment'. After one federal staffer creates a mashup that uncovers millions of dollars lost to Medicare fraud, Wolf Blitzer asks, 'Why doesn't every government employee have this at his or her disposal?' President Obama asks: "Is this the technology to connect-the-dots?"

3. Gartner turns back time. Realizing Enterprise Mashups went from Cool to Useful, Gartner puts Enterprise Mashups on their 2011 'Technologies to Watch' list. And using their previously-unknown time-shifting powers, they also retroactively insert it into their 2010 Watch List.

4. Enterprises shift from 'Me' to 'We'. It's no surprise that the key to increased employee productivity is to reduce cost and time needed to make important decisions. In 2009, there was much talk about "customized" data and widgets so users could make decisions faster, aka 'Me.' In 2010, the focus will be on collaboration and group decision, aka 'We.'

5. Google gets into the auto business. Google launches a solar car company called Google Cars which quickly becomes known on the street as 'Gars.' Gars aren't for sale; they are free to use. Gars are equipped with a 360 degree camera used for Google Street views and powerful WiMax Mesh antennas. Both inside and outside are live ads based on Google's new Geospatial-based ad auctioning system. You reserve a Gar for use for a period of time and pick it up at one of the 'Garplexes'. Your reservation priority is based on your Google points which are in turn based on how much time you spend with other Google products. Whether it's Google Droid, Gmail or Google Docs, you're always racking up points and earning the right to drive a Gar.

6. The BI industry learns to copy but forgets how to read. The big BI vendors see the impact agile, self-service technologies (like Enterprise Mashups!) are having on their customers and launches an all out assault trying to minimize mashups as a 'feature' of their massive BI systems. Unfortunately, they never read the Innovators Dilemma and fail to see that they are overshooting the market and Enterprise Mashup vendors are silently taking away BI Mashup market-share.

7. TIME Magazine names 2010 the 'Real-time' Year. Enterprises want their information as fast and as quick as a Google web search. Everyone knows it (but maybe the BI guys) and in recognition of this TIME also names Speedy Gonzalez as Person of the Year, although they say he 'isn't as fast as he used to be'.

8. Oracle acquires 9,782 more companies. This prediction doesn't take much insight, truthfully. Unfortunately, one of these 9,782 companies went out of business three years ago and another they already acquired two years earlier. I also predict Larry Ellison is not pleased with this.

9. Microsoft sells more SharePoint. (Yes, I know, I am prescient.) In 2010 Microsoft sells more licenses than all other software from all other software vendors combined. This drives Gartner to release the 'SharePoint Magic Quadrant', only to later realize Microsoft is the only one on it. Bill Gates is quite pleased with this and Larry Ellison is not.

10. The DIY folks form a union. In 2010 Generation Y-ers form the 'Open Self-Service Alliance' touting 'We can do it ourselves,' demand that IT support them, rewriting the 20/80 equation.

That's my top 10 or 2010. Love them, hate them, critique them, but remember where you heard them first. Good luck to all in this tech-packed New Year.

The Greatest Gift Your IT Folks Will Ever Give You: Self Serve

2010-01-04T21:00:58Z

The holidays are all about giving. If I were the Chief IT Elf, I'd tell my department to give you LESS. About 60% less. Before you think me a Grinch or a Scrooge, let me explain why, in this case, less is actually more.

In most organizations today it is quite common that 80% of the data-manipulation and assembly work is done by IT and 20% is done by the business unit (typically in Excel). If you want a BI report, you put your report request in to the IT and if all goes well, you get your report back some time in the future. And, if you are lucky, you can adjust a few parameters in a dashboard or portal and maybe get a more 'personalized' version of that report. But you, as the business folk, remain a 'consumer' of the data and the majority of the work is done by the IT team. Anything you request has a good chance of ending up in the IT backlog folder.

If there is one goal for enterprise mashups, it would be to flip the 80/20 workload and have IT do 20% of the work (mostly the heavy lifting) and the business unit (users and business developers) do the other 80%. At risk of scaring everyone (IT and business alike), another way to say this is have the business "self serve." If you follow the analysts, you know they're all talking about "self serve" (such as Anthony Bradley at Gartner). I like the Wikipedia description for general purpose self-service:

"Self-Service is the practice of serving oneself, usually when purchasing items. Common examples include many gas stations, where the customer pumps their own gas rather than have an attendant do it (self-service gas pumping is illegal in New Jersey & Oregon); Automatic Teller Machines (ATMs) in the banking world have also revolutionized how people withdraw and deposit funds; most American stores, where the customer uses a shopping cart in the store, placing the items they want to buy into the cart and then proceeding to the checkout counter/aisles; or at buffet-style restaurants, where the customer serves their own plate of food from a large, central selection."

Basically, people do things for themselves. And in the world of software, our 'non-technical' business folks are getting more technical all the time. I see the business unit doing things that are akin to Excel macros and Facebook sharing. In other words, self-service is viable when it's intuitive work done using intuitive tools. I definitely know very few people have read a how-to Facebook manual before "friending" (I don't even think there is Facebook documentation). And while you may be saying Facebook is a consumer website and the enterprise is different, I'd ask you to think about Salesforce.com. I suspect a majority of SalesForce.com users just figured 'it' out without any training or manuals. To go a step further, I doubt Salesforce.com would be wildly successful if it required IT.

Unfortunately, I think we sometimes view self-service as an all or nothing principle. It isn't. Let me give you an example. If I go to a self-service yogurt shop, I put the yogurt in a cup, put on the cover, get a spoon, napkin and head to the cashier. The cashier swipes my credit card, puts the yogurt in a bag and I'm done. Notice that I'm not working the cash register, the cashier is. It's still self-serve but not 100% and that's the point. When we talk about self serve, we're not saying the business does everything, but we are saying they do a lot more than they have in the past.

And your favorite yogurt store isn't alone. Self-Service has become a core part of your grocery experience as well. Decades ago, you'd order your food at the counter and had it collected for you. Today we pick our own food but complete the transaction with the cashier. And now many grocery stores have self-service checkout lines. At the checkout, you scan each item then put it on the belt. The systems are smart enough to track the scans with the items moving on the belt. Once you're done, you swipe your credit card and bag the items. But the entire 'self-service' process is established by the grocery store IT folks. You only get 'self-service' because of their hard work.

Enterprise Mashups are a great example of the self-service principle in the world of data. How does this work, exactly? If you think of the lifecycle of a piece of information, it usually starts at the 'I need...' stage and hopefully ends with 'I have and want to share...'. Mashups can apply to every step in this lifecycle but enable a large shift in the primary owners of many of the steps. Here's a quick outline how Enterprise Mashup work is often organized between IT and the Business:

Main IT Tasks (the '20%')
1. Install, configure, setup security and governance
2. Publish 'core' data sources (such as Webservices)
3. Assign roles and permissions for functions and core sources
4. Create 'complex' mashups, widgets and mashup dashboards, when required

Main Business Unit Tasks (the '80%')
1. Publish 'personal' data sources (from Excel, news feeds, etc.)
2. Visually create mashups, mashup widgets and mashup dashboards
3. Share mashups, mashup widgets and mashup dashboards with others
4.Rate, use, and personalize mashups, mashup widgets and mashup dashboards from others

As you can see, IT does the 'infrastructure' work such as setup, configuration and security/governance, publishing the high value or 'core' data sources and the creation of the complex mashups. This becomes the core infrastructure on which the non-technical folks do their self-service, much like the folks who build/install the self-service checkout register at the grocery store. This infrastructure is absolutely critical to enterprise mashups. Without it, you might have 'self-service' but it would be chaotic, not secure, scalable, governed and sharable self-service.

Let's resolve to make 20/80 the new 80/20 for 2010. Have a Happy Holidays.

Killing Zombies: Debunking 6 Common Enterprise Mashup Misconceptions

2009-12-03T16:49:24Z

Misconceptions are dangerous. Like zombies in a bad horror flick, they take on a life of their own as they spread through a community. And they have an annoying tendency to persist.

When the "Enterprise Mashups" term was coined circa 2007, some pundits quickly called them a poorly-aimed marketing ploy to rebrand some older technologies that had less than stellar reputations. Others saw it as a feeble attempt to steal the cachet of the cool new "consumer mashups" that showed apartments for rent in Chicago or LA. Unfortunately, because of these early knee-jerk reactions, enterprise mashups suffered some early but persistent misconceptions.

I think it's time to officially fight back against the zombies, so here are my top six picks for most common misconceptions about enterprise mashups. Remember, these are simply the opinions of one zombie-hunter, albeit a rather experienced one:

✦ Misconception #1: Enterprise Mashups are browser based.
Even though browsers are good at creating stunning, interactive Web pages and Web applications, they are not efficient at processing and combining significant amounts of data. This is a job that rightly belongs to the servers. The reality is enterprise mashups are best (i.e. fastest, safest, easiest to govern) executed on a server.

✦ Misconception #2: Enterprise Mashups are 'data on a map'.
Because the earliest consumer mashups were XML data on a Google Map, many thought that a 'mashup' was synonymous with 'map'. In my experience, most enterprise mashups aren't maps but charts, tables or other decision-support style formats.

✦ Misconception #3: Enterprise Mashups don't really require security and governance.
I once heard a software vendor executive on an enterprise software panel say; "Don't worry about security and governance -- that stifles innovation. Let security happen later." (You could hear the audience gasp in anticipation of the other panelist's response which was, in effect, "I don't care what it stifles, without security and governance it ain't livin' in my data center.") In fact, security/governance are the very essence of what makes a mashup into an enterprise mashup. They must be as safe as any application behind the firewall or they are nothing more than a science project.

✦ Misconception #4: Enterprise Mashups are 'throw away'.
I admit to being one of the perpetuators of this pithy idea a few years back but I've come to the conclusion that thinking of a mashup as 'throw away' is a bad idea because it is short-sighted. I find most people hear 'throw away' and think 'disposable' or single-use. But mashups can be quite the opposite of disposable, as they are easily changed for reuse in different situations and can be used as the building blocks of subsequent mashups. That can make a mashup more valuable than any static, one-trick-pony application.

✦ Misconception #5: Integration and Enterprise Mashups are the same thing.
There were some, particularly those working hard on SOA efforts, that said, enterprise mashups "sound like an ESB" or "are just more integration middleware." While I agree with Anthony Bradley at Gartner that some organizations are using enterprise mashups 'to clear out their application integration backlog more quickly and at less cost,' when in fact SOA-driven integration efforts and enterprise mashups complement each other well. And I'd say that I still see user-driven, user-facing decision-support and situational awareness solutions as the predominant enterprise mashup applications.

✦ Misconception #6: Enterprise Mashups are completely user-created, without IT.
Every few years some software trend blasts onto the scene claiming it will "put an end to traditional software development." Due to some early marketing (from a large software vendor that shall remain nameless), this was one of the qualities associated with enterprise mashups. But Enterprise Mashups access data inside the data center and IT owns the datacenter; as a result, IT is the one who must integrate the enterprise mashup platform into their organization's identity management, security and governance mechanisms. In effect, they build the expressway on which your mashups run smoothly.

Today, I think most would agree that Enterprise Mashups have a well-earned reputation as a serious enterprise technology. Both business and IT are welcoming enterprise mashups into their organization as a way to address their respective real-time decision-support and integration backlogs.

No zombies were hurt in the production of this blog.

Ending the Reign of the Swivel Chair: The Biggest Business Problem Enterprise Mashups Solve

2009-11-18T13:38:18Z

Interactive Data Corp (IDC) published a report a few years ago titled 'The Hidden Costs of Information Work'. In it they reported that 'searching for and analyzing information both consume 24% of the typical
information worker's time'.

IDC referred to these areas as 'relatively straightforward candidates for better automation'. I couldn't agree more. And with the volume of data doubling every 18 months, I'd venture to say this statistic is getting even worse. We're not getting less silo'ed, we're actually becoming more silo'ed. With all the mergers and acquisitions and SaaS offerings we've heard a continuous flow of horror stories of multiple systems managing everything from accounting, marketing and sales to customer support.

In short, we're not getting any better at helping our knowledge workers do their job. Instead these critical thinkers resort to "swivel chair integration," going from screen to screen (or if you're in the browser, tab-to-tab) copying and pasting data from one system to another. Not for data entry, but rather lookup and correlation to turn the data into information that supports their decision-making.

Of course, if these source systems were integrated, they wouldn't have to "swivel." But the reality is the systems they rely on are not integrated and with ever-shrinking IT budgets they never will be. And as IT gives these problems less attention, I think these important decision-makers are getting squeezed a little more every day, with more and more of their critical 'decision time' being replaced by simple 'gather' time.

In fact, I believe that silo'ed systems are so ubiquitous that many organizations have simply given up trying to integrate them. But our knowledge workers still have to make decisions based on all this data, so what do we do? Luckily, silo'ed data needed for informed decisions are a sweet spot for new, '2.0-style' information technologies like enterprise mashups. Enterprise mashups are tuned to easily and quickly gather data from many systems and presented in order to allow for real-time decision making. Here is a real-world example that follows the basic enterprise mashup pattern (extrapolated from a popular 'Data Center Mashup' video by Steve Graham, a major contributor to Apache Axis and now a Software Architect at the University of Chicago):

Problem: The IT Support team spends a significant amount of time analyzing software errors, trying to determine if the source of the problem is hardware-related, software-related or an end-user issue. The team needs better insight into the status of hardware and software assets on the network to help correlate these assets with the error tickets. The issues may be reported or discovered by the users, via SNMP traps, in application or hardware logs, or a combination of the three.

Example Data Sources:
1. HP Operations Manager: for gathering SNMP Traps for network monitoring
2. BMC Remedy Service Desk: for managing trouble tickets and incidents
3. Custom application #1: with all hardware server related information and IPs
4. Custom application #2: with all software assets and server deployment information

Decision Time Frame: As quickly as possible.

Impact: The longer it takes to isolate the problem, the more time, money and opportunity will be lost. Depending on the 'mission criticalness' of the software, the impact can be severe.

Solution: Real-time enterprise mashups tie directly into all four data sources, with a mechanism support engineered to locate information via IP address, server name, software system name and trouble ticket numbers. They then use information to filter and correlate from these sources and present information via a dashboard that shows the correlated network, hardware and software errors associated with each trouble ticket.

If you look at the above description and had to sum the problem up in a short statement, you'd probably say something like "The support engineers need data from multiple disparate systems to solve their problems. Since these systems are not integrated, they will have to do a lot of manual work to analyze the data across multiple systems. Let's make sure they all have swivel chairs!"

Hopefully, you get the idea. Enterprise mashups are a great way to solve the age-old integration-by-swivel-chair problem. Let's let the swivel chair return to its 'trusty but boring office companion' status.

Got Mashup Security?

2009-11-11T21:23:19Z

"Do you do security?" As a CTO, that's inevitably one of the first questions I'm asked when it comes to enterprise mashups. I sometimes feel compelled to respond with "Do you do security?" but I usually restrain myself. Unfortunately, a simple "yes" is insufficient as security without context is irrelevant, misleading and even downright dangerous.

To be precise, I don't think folks who ask this question are concerned about the security of the enterprise mashup itself. Instead I think they are more worried about the security capabilities of the enterprise mashup platform (EMP) and how it connects to their mashup data sources and security infrastructure.

Since I've heard this question repeatedly the last few years, I've had the chance to break the subsequent discussion into four areas. Every organization who is mashing will have to conquer these issues, whether it's 1 mashup or 100:

1. 'Plugability': Can the EMP plug into my existing identity management system?
Companies are [rightfully] looking for simplicity. What IT professional wants to bring in a product which creates yet another credential repository? None. The holy grail should be one seamless Single-Sign On (SSO), with every third-party solution (your EMP included) connecting directly to current systems, whether they are LDAP, Active Directory, PKI or even a proprietary system. And your EMP shouldn't be an exception to this rule.

2. 'Brokering': Can an EMP manage additional mashup services that require other security mechanisms?
Taking a pessimistic view of things, any 2 data sources you mash together will utilize 2 different security schemes, including old and new techniques like NTLM, Basic Auth, One-way SSL, Mutual SSL, WS-Security and even some proprietary mechanisms. Enterprise Mashups Platforms simply can't dictate security models to the systems they are mashing. Imagine asking for access to data sources and imposing your security model on them; after they tell you to 'take a hike,' you'll be in same position before you asked: without the data.

3. Internal Credential Passing: Can the EMP pass credentials to source systems inside the firewall?
This one is simple (yet surprisingly difficult at times). You want the same user's credentials to be used for authentication to all of the internal mashup data sources. In other words, if 'Jane Smith, Employee #4342' is executing a mashup of SAP and Oracle, you want 'Jane Smith' to be logged into SAP Payables and Oracle Payroll so that they: 1.) provide contextual data (i.e. data that is returned from these systems is data that Jane Smith should be seeing), 2.) user-specific, synchronous audit trails.

4. External Credential Passing: Can the EMP pass credentials to source systems outside the firewall?
This is the thorniest of the four scenarios. In spite of the many 'free' data sources on the Internet, many external application systems require user credentials to access their systems, just as internal systems do. But these external systems don't have to conform to your corporate standards, so the required credentials may be different from the credentials used to authenticate into your internal systems. The needed credentials may be as simple as user name/password or more complex such as, SAML attribute assertions. Whatever the requirement of the external services, the EMP must propagate the required credentials without a barrage of login requests to the user executing the mashup.

If your Enterprise Mashup Platform can handle the four scenarios above, then the answer is "Yes, we do security...and this is how."