Data Integration Blog

CleanUp! You first hear these words as a kid from your parents. Clean Up! When you hear this you usually know you’ve made a mess. Clean Up! This is what you shouldn’t be hearing, or, for that matter, thinking, in regards to your company’s data. Or, at least, if the prospect ever crosses your mind, it shouldn’t look as nasty and unpleasant as it used to in your childhood. =)

But nonetheless, clean up you should. If your source systems and the customer leads are a mess, of course. The obsession with clean data is only justified in this world of Business Intelligence, where looking at the picture as a whole and thinking big is not an encouraged, yet infrequent practice anymore, but a requirement.

One of the key elements to having your data clean and having a global view of your organization’s lifecycle is data migration. Wise data migration with an appropriate strategy and the right tools, not the sort where you splash money and remain in the same spot you started.

So how do successful companies approach this process? Simple, really. After you're solid in your decision to move you data, determine exactly what you're migrating and what it'll take. That is, identify the time, costs and human efforts that might be associated with this. It also helps to reherse. Unfortunately, not all integration vendors offer quality simulation/visualization tools, but that''s not too hard to fix.

Moving data around like that can also become a great opportunity to improve your data quality. That is, by checking its accuracy and consistency before sending it on to the new system.

Data quality problems cost U.S. businesses more than $600 billion a year, according to The Data Warehousing Institute. (14 Dec 2007)

How does that happen? How come companies are losing this much money and not even realizing there's a way to save up?

Again, the major problem usually lies in the source the data is received from and the way it is processed (if processed at all) before it enters the database/warehouse. Then there's also the currency of data, its accuracy, completeness, relevance and consistency.

When dealing with leads, a few of these problems can be avoided by using Online Lead Generation (OLG) techniques instead of buying from a big data provider with huge outdated databases. A recent Christopher Petix's article titled Marketers Should Avoid 'Dirty Data' discusses the major benefits of OLG as opposed to buying from data providers. Here are a few I thought really make OLG technique stand out:

- collecting new data for every campaign
- ability to set parameters for data collection
- consumers fill out information related to their demographics AND specify their interest in a specific product or service
- strict data cleansing processes, double-filtering

All of the above ensure a hugher degree of relevance, and, as a result, really help you save the money you'd waste otherwise on reaching out irrelevant leads.

So called "dirty data" not only wastes contact center agents' time, it also wastes a marketer's budget--and optimizing budgets is crucial in the current market conditions.

That's where you realize that data quality could be taken care of in advance, but you're stuck with your dirty data and need to deal with it (and consequently spend more).

Data quality has the ability to save marketers a considerable amount of money and is a completely transparent process.

What could be simpler?

With the speed on-demand solutions appear and gain popularity and compete with the old-school desktop systems, you'd naturally think customer data integration solutions have long become a usual thing. That's why it's a bit shocking to see the recent Forrester Research report only 2% of companies surveyed manage to achieve an integrated view of a customer data, while 92% claim this approach to be either "critical" or "very important".

Turns out, building bridges between web and desktop is still a problem, and we're only left guessing whether it's the loss of data, resources, or time that the companies are scared of.

But then there are enough pre-built solutions out there that won't take months or weeks of your time, and that you can use and be sure that nothing gets lost. Everything is simple, couldn't get more simple, in fact. Then why make it sound so compicated and scarry?

Just know what you need out of your CDI solution, and find one that would fit your requirements. Also, it helps to actually know what customer data integration is and enlightening your staff as to why the company need it. In this context, Jill Dyché's seven tips to get companies ready for successful customer data integration implementation might be useful.

ETL - Extracting and reading data from the original source, Transforming it to suit your business, that is, cleansing and formatting it, and finally Loading(sending) it to your system/database/warehouse

So, ETL tools. Make one of your own or buy one from a trusted vendor? What's best for your company?

Before jumping into anything or rushing to a popular vendor, analyze the degree to which an ETL tool will benefit your company and wether it might be wiser to build one of your own.

Building a customized solution of your own implies hiring technical staff for that purpose, but you have a greater chance of your final solution to be simple and match just your business needs.

Purchasing an ETL tool you are more limited by the market offerings in terms of customization. To add to that, you are likely to face the complexity of educating your in-house staff, which might take a load of your company's time and resources. Regardless of this, however, most vendors still resort to purchasing ready-made solutions. How come? Well, try asking yourself the following questions and you might actually come to the same decision.

1. What are your goals, why do you need an ETL solution? Is this a one-time procedure with limited conditions or are you planning to make it a part of your organization's structure and strategy? What do you want from your ETL tool? Specify priorities.
2. How much can you spend on your solution? How much do you want to spend?
3. How many data sources are you working with, and what kind are they? What functionality do they already have that might be helpful at the extraction/transformation.
4. How much time can you painlessly allow for the transformation process?, for your entire ETL process?
5. How much human resources and time can you dedicate to this project? (don't forget about education)
6. If you decide to build your own solution, who is going to educate your staff? Are you competent enough in the process (etl, warehousing)?
7. Just how many ETL experts do you have in your company and how do you estimate their potential and skill? Are they replaceable?

Just these 7 for starters...

As much as companies are talking of committing to Business Intelligence principles in their daily work, the concept of BI still seems too utopian and vague to be successfully implemented throughout an enterprise. It's probably not that the definition is vague, it's that the practical side will differ a bit depending on companies' needs.

The one thing that is more or less universal and requires the utmost attention in all cases is data quality.

Whether your data is already 'dirty' and needs to be reviewed on regular basis, or whether there is no systematic process for checking it within your company, sooner or later you realize that something about your data needs to be fixed. Some companies prefer to conduct regular automatic check ups, others choose to apply filtering techniques before the information even enters internal databases, one way or another, enough solutions already exist to help you make that first step into the world of BI and make it right.

One of BEYE bloggers recently posted his list of the top ten things BI lacks. Aside from data quality he singles out such foundational aspects of BI as the problem of structured and unstructured data, valuation techniques, Predictive Analytics / Data Mining, technology limitations, simulations, on demand analytics, etc. The the list will vary slightly from one company to another, but making one and working towards perfecting your Business Intelligence strategy through it is certailny helpful. No one can tell you how rewarding it is, you can only feel it for yourself while gradually putting "taken care of" or "implemented" next to each item from the list.