What is Data Unification?

Editor’s Note: This post was originally published in September 2023. We’ve updated the content to reflect the latest information and best practices so you can stay up to date with the most relevant insights on the topic.

To those unfamiliar with the process, data unification may seem insignificant. After all, it can’t be that hard to unify data, right? Wrong! Data unification is an incredibly complex process and one of the biggest challenges facing organizations today.

What is Data Unification?

Data unification is the process of ingesting data from various operational systems and combining them into a single source by performing transformations, schema integrations, deduplications, and general cleaning of all the records.

The Challenges with Data Unification 

To understand the challenges with data unification, think about all the different systems and tools used at your organization. Each one captures data differently. Now imagine trying to combine all of the data across those systems and tools into one master source. This process is incredibly difficult to achieve at scale, especially when it involves hundreds of thousands of datasets.

To give you a better idea of what this process entails, here’s a high-level breakdown of the data unification process from the viewpoint of Michael Stonebraker, Tamr co-founder and Turing Award winner:

• Ingesting data, typically from operational data systems in the enterprise.
• Performing data cleaning, e.g., -99 is often a code for “null” and/or some data sources might have obsolete addresses for customers.
• Performing transformations, e.g., euros to dollars or airport code to city_name.
• Performing schema integration, e.g., “salary” in one system is “wages” in another.
• Performing deduplication (entity consolidation), e.g., “Mike Stonebraker” in one data source and “M.R. Stonebraker” in another.
• Performing classification or other complex analytics, e.g., classifying spend transactions to discover where an enterprise is spending money. This requires data unification for spend data, followed by a complex analysis on the result.
• Exporting unified data to one or more downstream systems.

As you can see, unifying data is complex, which is why the vast majority of today’s organizations are facing a data unification crisis.

The Data Preparation Ecosystem

Because of this data unification crisis, there is an immediate need for organizations to have internal and external datasets that are agile and curated. And organizations that provide these types of datasets are part of the rapidly expanding data preparation industry that is expected to grow to $16.88 billion by 2030.

This explosive growth is largely because of the fact that data scientists spend 60% to 80% of their time cleaning and preparing data alone. Data preparation tools aim to greatly reduce the amount of time spent on data prep, with many companies incorporating the use of these tools into their data preparation workflows. 

Data unification is an integral part of data preparation and is an essential input to tools used by analysts and consumers, such as self-serve data prep tools and data catalogs. After all, these users can’t be expected to be productive and generate meaningful business insights without a foundation of trustworthy data, which data unification provides. 

Out with the Old: The Traditional Process to Unify Data Isn’t Effective

Legacy approaches to data unification typically include data preparation tools and rules-based master data management (MDM) solutions. Data preparation tools turn messy, inconsistent data into something usable by automating the processes to fix idiosyncrasies in the data that previously only humans could resolve.  

Traditional MDM solutions use rules to ensure critical business data is clean, curated, consistent, and up-to-date across disparate systems and sources. Both data preparation and traditional MDM are labor-intensive, requiring complex rules to unify data. These systems have a high upfront cost to develop, are costly to maintain, and do not scale as data becomes increasingly complex. As a result, companies often limit their data unification efforts to a select few high-value data sources.

In with the New: An AI-Centric Approach to Data Unification Is Highly Effective 

AI-native MDM overcomes the limits of rigid, rules-based MDM solutions by combining AI's efficiency and scalability with the human intuition and expertise you need to unify data and create trustworthy golden records. Not only is AI-native MDM dynamic, but it also enables agility and iterative development based on use cases that are important to the business. And when those use cases or the data that supports them changes, AI-native MDM can adapt, ensuring that the golden records it creates always reflect the most current and accurate version of your data. 

Further, new agentic data curation capabilities have the potential to revolutionize data management. Agentic data curation uses LLM-based AI agents to automate more of the data curation process by capturing and acting on the contextual insights needed to make confident curation and unification decisions. 

LLM-based AI agents have the capacity to replace traditional data preparation tools by intelligently cleaning, curating, managing, and refining the last mile of enterprise data—the part that addresses the idiosyncrasies and complex edge cases that are close to consumption and difficult to decipher—with minimal human intervention. By comparing outputs of entity matches and explaining the reasoning behind why records do or do not match, AI agents can provide the preliminary analysis humans need to determine if they trust the AI’s output or if they need to tune the model further before unifying records.  

Unifying Data to Identify New Cross-Sell and Upsell Opportunities 

A Fortune 500 global life sciences company was facing significant operational challenges related to their data. With customer data fragmented across more than 15 operating companies, each with its own CRM system, it was virtually impossible to gain a holistic view of customers. As a result, the company missed out on valuable revenue from cross-sell and upsell opportunities.

Using Tamr’s AI-native MDM, the life sciences company unified data across 17 internal and external sources—in less than six months. They also deduplicated over 10 million contact records, improved record completeness by nearly 10%, and delivered a 360-degree view of their customers across the business. 

As a result, the company enabled AI-driven use cases including:

• Next best action for sales
• Marketing personalization
• Cross-sell and upsell opportunities
• Customer journey analytics

Unified Data Improves the Provider Onboarding Experience

At CHG Healthcare, fragmented provider records lived across multiple, disparate systems, resulting in a frustrating, manual onboarding experience. Because their records were inconsistent and disjointed, CHG staff asked providers to provide the same information multiple times. 

CHG implemented Tamr’s AI-native MDM solution to help them overcome this challenge, and within a matter of weeks, they mastered more than 7 million provider records and reduced duplicates by more than 40%. Using Tamr RealTime, CHG also enabled real-time access to provider data, minimizing the risk of creating duplicate provider entries in their systems.

In addition, because their data is now unified, accurate, and complete, CHG can pre-fill paperwork for providers during the onboarding process, saving them time and eliminating frustrating, manual processes. 

Final Thoughts

According to the latest estimates, 402.74 million terabytes of data are created each day. And this number is only growing. Businesses need to unify data using AI-native MDM so they can make smart, data-driven decisions and compete in a global economy. You’ve heard the expression knowledge is power. For modern-day businesses, that knowledge comes from having complete access to reliable, unified, trustworthy data, and avoiding pitfalls along the way.