If there is one thing I hear from SOA/EA architects out there is that it's difficult to get the design, development, and deployment teams focused on the data. The real fun seems to be in the services and processes, and thus most work is concentrated there. The data, and integration strategies around the data, is something that most figure is there, will be there, and requires very little thinking and planning.

However, those who neglect the data, and the integration of that data with other systems, are going to find the bitter truth at some point. Data quality issues rise up first, then data consistency issues, and finally the lost faith in the IT system by the end users, and typically a the death spiral at that point. Time to redo things, and that's both expensive, best case...career limiting, worst case.

So, how do you make data integration a priority? It's really a matter of just looking at the business benefits, and thus the business case for data integration. Without a sound data integration strategy you're simply not going to be able to meet the requirements of the business, and can't get more clear than that.

Moreover, you need to consider data integration as something that's systemic to your architecture, and not just an afterthought. Back in the EAI days, it was all about adding a message broker between major systems, and life was good. It's was a Band-Aid at best.

These days it's about managing data from the source, to the target, including any binding to services as required by the SOA. You can't back your way into that, thus architects needs to do...well...architecture, thus a bunch of advanced planning to make sure that the data integration strategy is sound, and brings the value to the IT infrastructure, and therefore the business. In other words, it's needs to be an architectural priority.

"Mounting evidence suggests that in 2010, the hottest segment in BI (business intelligence) software will revolve around offerings delivered from the cloud, thanks to increased product sophistication, strained IT budgets and other factors.

A new IDC report finds the SaaS BI market will experience triple the growth of the market overall, soaring at a compound annual growth rate of 22.4 percent through 2013, although actual revenue totals will remain small compared to on-premise BI applications."

Good to hear, but 22.4 percent is a bit excessive if you ask me. However, I can understand why this would occur considering that budgets are tight, also SaaS BI is addressing a portion of the market that traditional BI technology has not addressed, such as front office operations. Those guys have not had the BI tools required to do their job, and many SaaS BI providers, such as Good Data and others, and looking to fill that slot.

Additionally, with the BI tool living out in the cloud you now have the perfect opportunity to mash up your BI data with lots of interesting third party data, such as your sales compared against census data, or sales growth as related to other data sets. You can have a bunch of fun with that, trust me.

To be honest I've been a bit bearish on SaaS and BI for the last few years. The existing SaaS BI tools had some baking to do before they are ready for enterprise prime time. That said, I now believe that most of the BI SaaS players have done a good job in addressing those needs, and you can count on SaaS BI being mixed in with traditional BI systems for the next few years, perhaps replacing those system in 5 years. It's all going to the cloud, right?

I loved this Information Management article by J. Brooke Aker talking about the links organizations can make between the semantic Web and MDM.

"MDM requires extensive understanding of business terminology and the context and importance of data. Furthermore, much of the corporate data MDM needs to capture and organize is unstructured, buried in emails, corporate Intranets and white papers.

A hybrid semantic Web MDM approach can solve these tough issues."

So why the semantic Web? As the author puts it the semantic Web can enhanced MDM solutions by locating and managing any duplicate data. Indeed, semantic Web technologies are able to identify and analyze customer data, purchasing trends, market deals and competitive information. Moreover, semantic Web technologies can uncover unstructured data not only from within the existing enterprise systems, but also from unstructured Web delivered data as well.

So, where is the fit? Semantic Web technologies can enhance MDM solutions by locating and managing duplicate data. "For example, if there are records of the same customer but filed with different names (Enterprise, Inc. and Enterprise Corp; Hank Brown and Henry Brown), semantics can assist MDM and identify the different meanings of the same customer and give true insight into the customer's information."

MDM seems to be like "going to the gym," for most enterprises. Everyone talks about it, perhaps spends money on it, but few are following through. Truth-be-told many of the MDM solutions out there that I've been briefed on are less than innovative, and perhaps learning to leverage standards such as the semantic Web, if they not have already, is the right path to MDM becoming a much bigger deal within the enterprise. We're not there yet.

Bob Evans over at Information Week created and discussed a list entitled "Global CIO: The Top 10 CIO Issues For 2010."   Which, BI was number 5 out of ten.   As I recall, that's falling a bit.    

"Your organization is probably awash in data--mountains of it, probably--but is there a corresponding volume of insightful information? And out of all that insight, are you able to gain advantage through foresight? Can you see around the corners and over the horizon? Or do you just arrive there and hope that what you've got matches what's needed in those unseen precincts? Business analytics and in particular predictive analytics will become enormously important in 2010, and CIOs who seize the initiative in turning insights into actionable foresights will have a huge advantage."

Right on.   The trouble is that most CIOs are more bogged down with the daily activities of running an IT shop that the concepts of BI really never come into the discussion, and the benefits are thus not known.       Moreover, BI is officially considered "old technology," where this like cloud computing are the sexy topic of the day.   So, what's a CIO to do?

First, there should be some prioritization within the ranks.   While Bob provides us with a nice list of things to consider, the things you actually fund within your IT shop should match up with your business requirements.  Call me crazy. ;-)     In many cases, BI is much more important than the other items listed.   Keep in mind you can't pick all 10, you may be able to do 2 or 3 in 2010, depending on your resources.  

Second, make sure to fund these efforts accordingly.    BI is expensive.   The consultants cost an arm and a leg, the technology is not cheap, and if there is not enough cash to lubricate the path to success, it's not going to happen.   You can't do BI on the cheap.

Finally, create metrics to monitor the results and the resulting benefits to the business.   Make sure you take time to pat yourself on the back, and thus find more money to improve your IT assets.

I really enjoyed Jill Dyche's recent blog posting entitled "Could Data Governance Help the War on Terror?"   In essence, calling out the recently break down in intelligence that failed to take the "underwear bomber" off the plane before attempting to blow the thing up over Detroit.

"Instead, the government should be addressing process issues. Indeed, the real conversation should be how to move forward. These questions should be asked now: 'How should we bring identifying data together? What are the key sources? How should integration, access, and usage policies be formulated? What would a sustainable process look like?' Those questions aren't 'who' questions, they're 'how' questions, and they should be front-and-center in the national security conversation."

What's been occurring in the intelligence community is a movement to get data integration right within their own domains, but not consider integration between agencies.   Thus, they are missing key pieces of the puzzle in truly evaluating threats, which basically comes from patterns that are culled from data sources that span organizations.     Clearly, we don't do that well, if not at all if we're missing things that are that obvious. 

Data is key here, and perhaps more the answer to the problem than making me put away my iPhone an hour before landing.     What's most bothersome about things like this, and even 911, is that the data was there, however no mechanisms existed to make good use of it.   In short, there was no data government or management in place.

Not to blame technology entirely.   The culture of "not sharing my toys," in this case data, that exists between agencies is getting a bit annoying as well.   

To be honest, this is easy stuff.   The aggregation of information from many organizations is something that businesses do daily in support of profits.   Let's give it a shot in support of safe travel...shall we?   

The amount of data that systems are collecting on you right now is amazing. Google knows your Web habits, Facebook knows your friends, education level, and relationship status, Twitter knows when you had breakfast, lunch, and dinner, and Amazon knows who you purchased gifts for in 2009. As separate piles of data they are really not providing much information unto themselves, but when brought together can form clear patterns around who you are and what you do.

Up until recently this data had little value. However, those who make money selling this data, typically all of the major social networking sites have learned to keep track of you. After all, they have to get paid somehow, and their data is their asset. They, of course, have privacy policies, and most won't give up your individual information without a court order. However, this is not about that. It's about leveraging very sophisticated BI systems to put together all of the bits and pieces of data about you, and a surprising very clear picture of you. This would include preferences, buying habits, eating habits, vices, your relationships, income, taxes paid, ..., in essence things you may not have known about yourself, but others do.

What's interesting about this is that the information until itself existing out there does not cause privacy problems; it's the analysis of the information holistically by a BI system. Thus, while BI systems are the single best opportunity for understanding your market, they are also the single largest threat when considering privacy issues. I guess you have to take the good with the bad.

First, data management issues in most Global 2000 companies are nightmares, typically. The fact is that many heterogeneous databases typically exist, and there are no mechanisms or processes in place for controlling all of that data. Thus, a data element is altered in one database breaks 20 things and no one knows why. Typically, this is under control of a single human within the company, and that can be a bit dangerous.

Second, security issues are typically afterthoughts, and always require a good data governance program. It does not make sense to hide systems behind passwords, if the underlying data structures are easily accessible and altered without adherence to any policy. You can't lock the front door and think your safe, if the backdoor is wide open.

Finally, the movement to SOA and cloud computing requires strong data governance processes and technology. You can't manage services without underlying data governance; else things will be breaking quickly. Indeed, the entire sets of layers need to be governed, from the data, to the services, to the processes.

More consolidation in the data integration world. Most of the data integration companies have been around for a long time, and they are either huge or small. In 2010, the larger players will surely buy up the smaller players, before the prices get too outrageous as the market recovers. The good news is that most of the smaller players have a good customer base and tasty technology.

Cloud computing drives much of the growth of data integration. While we've not see this as of yet, the continued interest in cloud computing will certainly drive a lot of data integration in 2010. The fact of the matter is that enterprises don't think about data integration until they have already moved to the cloud. That's like building a house without a driveway.

Data integration standards around vertical spaces continue to emerge. Nothing new here, but in 2010 there will be more of a focus on data integration within vertical spaces, including health, finance, and government. Metadata and security standards will lead the way.

Security vulnerabilities will be exploited. Needs to be some bad news as well, and I believe in 2010 we'll have a few instances where bad people exploit some vulnerability within data integration technology. Thus, we'll see some data leaking out, but not much worse than the stolen laptops and data theft via a USB drives we've seen in the past.

So, how do you sell Data Integration? Let's look at a few key concepts, including:
1. Shining a light on existing limitations.
2. Creating the business case.
3. Creating the execution plan.
4. Delivering the goods.

Shining a light on existing limitations refers to the process of admitting how bad things are. This is difficult to do for most architects, because this can expose you to criticism. You're in charge of keeping things working correctly, and now you have to explain why things aren't working as they should or could. If it's any comfort, the architecture within most Global 2000 companies is in need of fixing.

Creating the business case refers to the process of actually putting some numbers down as to the value of the Data Integration to the enterprise or business. This means looking at the existing issues (from the previous step), and putting dollar figures next to them. For instance, how much are these limitations costing the business, and how does that affect the bottom line? Then, how will the addition of Data Integration affect the business -- positively or negatively?

Creating the execution plan refers to the detailed plan that defines what will be done, when, by what resources, and how long. This is basically a project plan, but most people will find that the systemic nature of Data Integration requires that a great deal of resources work together to drive toward the end state. Leveraging and managing those resources is somewhat complex, as is the project management aspect of Data Integration.

Delivering the goods just means doing what you said you would do. Execution is where most Data Integrations fall down. However, if you fail to deliver on-time and on-budget, chances are your Data Integration efforts won't continue to have credibility within the enterprise, and future selling will be impossible. So, say what you'll do, and do what you say.

Selling Data Integration is more of an art than a well-defined process, as you've seen. It really requires a certain degree of understanding, including the technology, the business, and the culture of the enterprise. Most important to the project, selling needs to be followed up with delivery and the value. That's the tough part.

The issue here is not that the Gartner report is incorrect, it's that specific rankings around the good or bad aspects of a data integration tool has very little to do if the tool will be productive for you, or not. I'm finding too many end users looking at the 4 boxes, versus looking at their own requirements, and thus select the wrong technology for the data integration job, but always attaching the "Magic Quadrant" to the last slide in the presentation to justify the selection.

The core issue is that data integration is such a wide range of patterns that it's very difficult to address data integration in the narrow, looking at the data integration concepts and generalities. There are many different types of data integration solutions, such as abstraction, replication, real-time integration, near time integration, batch,..., you get the idea. Thus, you'll need to match the right solution to your requirements. Make sense?

It's okay to use the "Magic Quadrant" for general guidance, but the tools you select need to align with your requirements, no matter where their dot sits on the quadrant.

"Business Intelligence Applications: There are a lot of business intelligence applications, but predictive analytics and enterprise performance management applications have been one of the few bright spots during the economic downturn, mostly because they allow IT organizations to quickly add value to years of investments in ERP applications that have increasingly left the business feeling skeptical about IT investments."

What was surprising to me is the number of people who consider business intelligence (BI) an "old technology" pointing to the interest in data warehousing that started in the middle 90s. However, today BI applications are providing some real value we'll beyond slicing and dicing data warehouses and data marts. In short, we can see what's going on and react to it. We're focused on operational effectiveness as well as decision support.

As Mike points out the perceived value that everyone is experiencing around BI is the externalization and visualization of information that has, until very recently, been locked inside of stovepipe systems. Today we are getting much better at linking too those systems and peering in at "near real time data" that actually helps us do our jobs. What a concept.

Considering that pattern of architecture there are two basic types of ontologies, horizontal and vertical.

Horizontal ontologies are general in nature, such as space-time relationships. These are common ontologies that span multiple domains, are not applicable to any single vertical space and provide a mechanism to organize and standardize information content. We've employed this type of ontology for years in the form of object models, hierarchies, taxonomies and, in many cases, XML vocabularies.

Vertical ontologies, which also incorporate features from horizontal ontologies, are domain-specific, such as natural languages for healthcare or financial services. Vertical ontologies not only define data in terms of semantics native to a particular vertical industry, they also contain rules and formal computer languages that can perform certain types of runtime automated reasoning. This means we understand the meta data and have logic bound to the meta data as well.

The use of vertical ontologies, which extend the capabilities of horizontal applications, is where the most value exists. As we learn to define these ontologies as common frameworks for specific business requirements and define the reuse of such frameworks applicable across multiple like-domains, we also learn to apply languages and reasoning techniques. Ultimately, this provides repeatable information formats, rules and logic that, in turn, provide data integration architects with the ability to leverage existing solutions rather than form them from general-purpose middleware and application development technology.

All valuable when considering understanding data, and the use of data, within our problem domain.

Clearly, others see it as a trend as well, including Rob Barry over at Tech Target.

"The company has pulled together all of its relevant data management offerings into a single platform. Though the release is a big step for Informatica, some industry experts see it merely as part of a well developed trend."

Informatica 9 will provide an enhanced data integration platform, combining products in data integration, data quality, B2B data exchange, application information lifecycle management, complex event processing and cloud computing data integration. This data integration meta stack is really the direction of the data integration business, with a focus on a single layer that can provide traditional ETL-type data integration, data quality services, and most importantly the ability to address complex data sets using a single point of view, or data abstraction.

I've been a huge advocate of data abstraction, or the use of well defined data services to deal with the complex and heterogeneous physical databases that are a large part of the Global 2000 and government. A good data abstraction layer is able to solve many problems, and is a key component of an SOA. If you don't manage your data assets correctly, than you have no chance of doing much else right within and architecture.

Keep a close eye on this trend.

Keep in mind that the core value of Web APIs are their ability to produce and/or consume information or functional behavior, machine to machine, server to mashups, or server to any Internet-connected application. Thus, you can mix and match information and behavior from any number of sources to form a business solution. Some of these sources you own, perhaps some you don't. These are typically non-visual services, and the interaction is happening behind the scenes. The power of this concept is the fact you don't own nor host these services; many are free and or at low cost to you. Thus, the value proposition is speed, agility, and efficiencies.

So, how do you find the thousand of APIs out there scattered among hundreds of API providers? Just as we needed a mechanism to find Web content back in the 90s and search engines appeared, new directories are appearing, and will continue to appear, allowing you to find the right API. Early leaders include the Programmable Web API directory, with hundreds of APIs already listed, and more added each week.

What's cool about this is that human-readable directories won't be the direction here, but instead shared repositories of APIs/services. In other words, the ability to link to a shared database that contains information about thousands of available APIs/services, and have that repository be part of your existing private service repository. Thus, you have a private repository of services you own exclusively, or are private, and also include APIs/services that are public.

Considering this within the context of the emerging notion of SOA, you're able to break your existing systems down to a collection of services that can be leveraged within business processes to create business solutions. Along with those services, you can also add thousands of public APIs/services for things such as demographics data, mapping, social networking, messaging, etc., and be able to mix and match these APIs/services, private or public, to create some killer business solutions. This is the emerging concept behind the enterprise mashup.

The critical success factor here is the ability to go from a state where the APIs are widely scattered, deployed differently, and are difficult to find, to a shared infrastructure where the APIs/services can actually be discovered and leveraged as if they were local. In many instances I suspect they will be even more reliable than their local, private counterparts when considering that there will typically be more investment in the infrastructure around public APIs/services than services that are intra-enterprise.

While we are waiting for a common directory/repository, or while we're awaiting the existing directory/repositories to refine them, it's still a good exercise to learn how to leverage external APIs/services within your core business applications, even if it's just as a proof of concept. The fact of the matter is that our ability to find and leverage the right APIs/services quickly, and leverage those services within your core business processes could be the secret sauce that allows you to out-innovate your competition.

Interesting article by Korhan Yunak discussing the creation of a BI strategy within the enterprise, and the core drivers behind it. In essence, while there are many surveys that show that BI is a top priority year after year, "and almost every company invests in some sort of BI initiative." However, very few are talking about their BI programs has a "huge success" or something that dramatically improved the performance of their company?

I've been hitting on this very topic a lot in the last month. Indeed, BI, while an interesting and valuable concept is not getting the implementation and planning resources it requires and thus it's largely tanking within most enterprises. Organizations are implementing a data warehouse and calling it good, but a data warehouse is not necessarily BI, and thus the value may not be there.

The reasons are all over the board, but the core issue is, as Korhan states:

..."there is a big difference between having a data warehouse that is obviously disconnected from the decision making process and having a BI program that is integrated into all aspects of the business and decision making process."

The core issue is lack of focus on the "B" in "BI," or business. Thus, the BI systems fail to reach the core business benefit that the stakeholders are expecting. I know many that are so proud of their very expensive and very sophisticated data warehouses, only to find that they are not solving any significant business problems. Why? Business was never involved.

..."business intelligence is for business, which means that a BI strategy should be adopted by business stakeholders. Stakeholders need to have a good understanding of what the purpose of the BI strategy or program is. To build this common understanding, the first thing to do is take their needs into account at the very beginning. Go to them and ask what it is that business would like to achieve with the BI initiative or program. And then create a strategy document that describes all these objectives and aspects with no reference to any technical consideration."

Core IT continues to manage around hype rather than business objectives, and thus do not have much business benefit from rather costly IT efforts. You may want to review your BI program to make sure you're not these guys, most of you are.