We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

New Frontiers in Business Intelligence

Nari Kannan

Fixing the Second Root Cause of Bad Data Quality - Perpetual ETL Battles!

user-pic
Vote 0 Votes

Sometime ago I wrote in another blog entry - Five Root Causes of Bad Data Quality in Business Intelligence.

The Second Root cause of Bad Data Quality is the group of errors made during Extract, Transform and Load (ETL) programs.

ETL programs are very insidious in the sense that they work perfectly in theory but never in practice!

If you have the data in a data warehouse and you use a query to simply extract the data and do a report on it, you may not be getting that big a bad data quality issue! But that's rarely the case.

Almost in any large organization, reality is that you have many distinct SILOS of data rather than one monolithic source due to functional specialization of software.

So invariably you had to do Extract, Transform and Load (ETL) functions from one or more slios to get meaningful reports done. Or alternatively, you had to do ETL transformations to get data to a central data warehouse.

As if things were not complicated enough, organizations are increasingly using outsourced services and it may involve integrating data that comes from outside the organization.

The global Insurance industry is a classic example, especailly high volume transaction oriented ones like Automobile Insurance. They have to deal with individual repair shops that are local,parts suppliers that may be local and even the damage estimation function is outsourced these days.

So the Insurance company that pays for repairs to automobiles have to deal with data flowing in and out of many of these parties' internal systems.

So without ETL you cannot have any of these function properly.

When you are dealing with outside sources of information or even internal silos, formats change without everybody being notified and ETL programs start creating bad data through transformations that may no longer be valid.

If the ETL program breaks, you would know about it and you can go fix it. But what if the changes are subtle, the ETL program does not break and only puts out bad data?

Another link in the Bad Data creation cycle that needs to be paid attention to.

It is a capital mistake to theorize before one has data. - Sir Arthur Conan Doyle

Nari Kannan's blog explores how new approaches to business intelligence can help organizations improve the performance of business processes--whether these processes are creative or operational, internally-focused or customer-facing, intra-departmental or across functions.

Nari Kannan

Nari Kannan started and serves as the CEO of appsparq, a Mobile Applications development company based in Louisville, KY with offices in Singapore and India. Nari has over two decades of experience in computer systems development, translating product and service strategy into meaningful technology solutions, and both people and product development. Prior to this, he has served as both Chief Technology Officer and Vice President- Engineering in six successful startups, two of which he co-founded. He has proven experience in building companies, engineering teams, and software solutions from scratch in the United States and India. Prior to this, Nari started Ajira Technologies, Inc., in Pleasanton, CA, where he served as Chief Executive Officer for more than six years. While at Ajira, Nari was instrumental in developing service process management solutions that modeled, monitored, and analyzed business processes, initially targeting the Business Process Outsourcing (BPO), Telecom, and Banking verticals in India, and Finance, Insurance, and Healthcare verticals in the United States. Prior to this, he served as VP-Engineering at Ensenda, an ASP for local delivery services. He also served variously as Chief Technology Officer or VP-Engineering at other Bay-Area venture funded startups such as Kadiri and Ensera. He began his career at Digital Equipment Corporation as a Senior Software Engineer. Nari has a long involvement with Customer Support and other customer facing processes. At Digital Equipment Corporation he was involved with their 1800 person customer support center in Colorado Springs, Colorado. He was tasked with coming up with innovative tools to help customer support people do their jobs better. He holds a U.S patent for a software invention that automatically redirected email requests for customer support to the right group by digesting the contents of the request and guessing at which software or hardware support group is best equipped to handle it. At Ensera, he led a 45 person team in developing an internet based ASP service for handling auto insurance claims, coordinating information flow between end-customers, Insurance companies, Repair shops and Parts suppliers. Ensera was acquired by Mitchell Corporation in San Diego. Nari holds a B.S. degree in Physics from Loyola College, and an M.B.A degree from the University of Madras in Madras, India. He graduated with a M.S. degree in Computer Science from the University of Massachusetts at Amherst in 1985. Contact Information: Nari Kannan. Email: nari@appsparq.com Mobile: 925 353 0197. Website: www.appsparq.com View more .

Subscribe

News Feed

Recently Commented On

Monthly Archives

Blogs

ADVERTISEMENT