What is Agentic Data Curation? Perspectives from Tamr’s Co-Founders

The ways in which organizations master their data is changing fast. Historically, traditional rules-based master data management (MDM) was the solution of choice to unify and clean enterprise data. As proven over decades, however, a rules-based approach is complex, costly, doesn’t scale, and only gets organizations 40-60% of the way toward actually achieving fully clean, trustworthy data. 

Machine learning-based solutions like Tamr make a big difference, getting organizations 90-95% of the way toward accurate, unified data, a feat rules-based systems alone can never achieve. But it’s the “last mile”—the 5-10% of data that remains unresolved after the mastering process—that is the most difficult and time-consuming to resolve, requiring a higher level of knowledge, precision, and data preparation. That is, until now.

There is now a promising new approach that Tamr is pioneering that enables organizations to shorten this last mile of curation of data closest to consumption: agentic data curation. 

What is Agentic Data Curation?

Agentic data curation is a new concept that has the potential to revolutionize data management. Using LLM-based AI agents to automate more of the data curation process, agentic curation captures and acts on the contextual insights needed to make confident curation decisions around specific scenarios. 

It’s a well-known fact that the last mile of enterprise data mastering—the part that addresses the idiosyncrasies and complex edge cases that are close to consumption and difficult to decipher—is generally the hardest to resolve. And it usually requires expert human involvement to do so. But with the introduction of intelligent LLM-based AI agents, organizations now have the capacity to clean, curate, manage, and refine this last mile of data with minimal human intervention. 

Imagine having a small “team” of AI agents that are purpose-built to fill in the full picture of an entity by adding and standardizing values for fields such as address, industry, website, company type, and more. LLM-based agents can use the identity of an entity (e.g., company name) to complement Tamr’s core ML-based data mastering workflow, using trusted resources such as a firmographic data provider like Dun & Bradstreet to identify additional information that helps to build out the record. Once the agents collect and verify this data, they can propose these values which humans then review and approve before they are committed within the database. 

Using advanced, agentic AI capabilities, AI agents can not only compare outputs of entity matches, but they can also explain the reasoning behind why records do—or do not—match. They can also deliver additional context and the preliminary analysis humans need to determine if they trust the AI’s output or if they need to tune the model further. 

Perspectives on Agentic Data Curation From Tamr’s Co-Founders

As leaders and academics in the industry, Tamr’s co-founders offer unique perspectives on the future of MDM and how AI agents will transform the market. 

To explore the concept of agentic data curation further, we talked with the six co-founders of Tamr. In these conversations, they offered their unique perspectives on the future of MDM and how agentic curation and AI agents will transform the future of MDM. 

Below is an excerpt from our ebook, “How Agentic Data Curation is Transforming Data Mastering: Perspectives from the Tamr Co-Founders,” featuring the perspectives of three of our esteemed co-founders. Download the full ebook for a deeper dive into our co-founders’ perspectives, including additional insights from co-founders George Beskales, Dan Bruckner, and Alex Pagan.

‍

Michael Stonebraker, MIT Professor of Computer Science and Turing Award Winner

‍During our conversation with Michael, he shared that when it comes to data mastering, machine learning is the only thing that scales. And it’s not going away. But he also thinks that large language models (LLMs) could play a role as well. After all, they are pretty good at ingesting natural language and spitting out SQL. But he also believes this question still remains: How accurate will the AI agents be when working with enterprise data? If we see 60% accuracy using an LLM-based AI agent on enterprise data, then it’s not ready for prime time. But if the accuracy increases to something higher than 90%, then things get interesting. 

‍

Ihab Ilyas, Co-Founder and CEO, hiddenweights

According to Ihab, agentic AI is giving us the opportunity to redefine “human in-the-loop.” AI agents can handle popular or familiar cases that are known but highly complex, which require humans to remain involved and maintain a level of control while still doing less. 

In practical terms, Ihab explained that if we take a look at the curation pipeline, it comprises three distinct processes: data prep, data mastering, and post-processing. AI can help at every point, which presents a great opportunity to rethink the end-to-end pipeline. For data prep, AI embodies a data scientist skillset providing capabilities such as pivoting, removing anomalies, and standardizing data. Within data mastering, it handles specific cases that require specific context. And in post-processing, AI supports answer generation, transformation, and generative AI capabilities through AI agents. With agentic AI, the data pipeline will become more accurate, more scalable, and more efficient, removing any excuse for inaccuracy. 

‍

Andy Palmer, Entrepreneur, Investor, and Tamr Chairperson

Andy shared that AI agents are poised to play a significant role in the mastering process, especially when it comes to self-service data preparation and data curation. Using agentic data curation, AI agents can bridge the gap between data curation and last-mile data preparation by automating the adjustments and preferences users typically make, ultimately reducing the amount of manual work. By observing the actions humans take in data preparation, AI agents learn about the idiosyncratic changes users apply and can then automate these changes, acting very much like a macro on steroids. And because agents will begin to automate some of the manual prep work previously done by humans, data teams will be able to shift their focus to spend time closer to the business. 

‍

The Risks of Agentic Curation

While LLM-based AI agents hold incredible promise, our co-founders advised that there are risks to consider. In the ebook, they also share strategies to overcome these challenges. 

1. Applying AI agents to your full data set ‍

AI agents are powerful, which is why it’s risky to turn AI agents loose on your entire data set.

2. Losing control of the agents—and your data

AI agents are a smart tool—but remember, they are still a tool. 

3. Hallucinations 

Agents will get things wrong, and it’s not always obvious when their insights are incorrect.

4. Diagnostic challenges

Understanding why the agent made a mistake —and knowing how to fix it—is very challenging.

5. User skepticism

Many users will be skeptical about the information or guidance an agent provides. 
‍

Putting AI Agents to Work 

At Tamr, we’ve always focused on delivering trustworthy, golden records for use in analytics and operations. And while an ML-based MDM solution like ours can deliver 90+% of the golden records an organization needs, automating the work to resolve the remaining 5-10% has felt out of reach…until now. 

Agentic data curation offers a new, promising approach to data mastering and data curation that enables organizations to improve data quality at scale while shortening the last mile curation of data closest to consumption. 

Tamr has a distinct advantage in the market, and we are poised to take full advantage of AI agents through our AI-native platform. More on this coming soon!