In his compelling April 6th piece in the New York Times, Steve Lohr asks the question: “If Algorithms Know All, How Much Should Humans Help?”
It’s a burning question, as more and more business, political, social and cultural decisions are being informed by machine-driven big data analytics.
It’s a substantial enough question for MIT to announce in the same week the formation of an institute — for Data, Systems, and Society — to address some of the question’s key elements
And it’s a question at the core of Tamr’s design pattern for data curation at scale: specifically, what’s the ideal human-machine collaboration in unifying and preparing vast amounts of heterogeneous data for analysis?
One approach is to give full agency, beforehand, to the algorithms, relying on the machine to automate everything. As Lohr writes, this might okay in “low-risk” categories like marketing.
But the stakes are rising as the methods and mind-set of data science spread across the economy and society. Big companies and start-ups are beginning to use the technology in decisions like medical diagnosis, crime prevention and loan approvals.
A natural solution for mitigating risk in these high-stakes fields is, as Lohr puts it, “close human supervision of an algorithm’s results.” Or, in other words, human review and approval after the machine does its work. The problem here? As Lohr describes it …
[H]uman bias. The promise of big data decision-making, after all, is that decisions based on data and analysis — more science, less gut feel and rule of thumb — will yield better outcomes.
We’ve taken a third way at Tamr — involving human experts during the machine’s process … not before it starts or after it’s finished. We call our approach “Machine Driven, Human Guided,” which Tamr Co-Founder Daniel Bruckner described in his recent Rise of the [Data Preparation] Machines post:
On top of state-of-the-art algorithms that automatically unify hundreds or thousands of data sources, Tamr provides a simple but potent user interface for matching, cleaning and understanding disparate data sets. When the machine can’t resolve connections automatically, it calls on “curators” — experts in the organization familiar with the data — to weigh in on the mapping and improve its quality and integrity. In the driver’s seat, a data curator can steer Tamr’s machine intelligence through any danger.
At Tamr, “Machine Driven, Human Guided” goes well beyond a pithy tagline [not that we have anything against pith]. It’s our founding philosophy, the idea that humans and machines should not just collaborate, but iterate on problems that can’t be solved manually or through automation alone.
In other words, algorithms v. humans is not an either-or decision. “Together” is possible, with the proper technology and process.
To learn more about Tamr’s machine-driven, human guided data unification platform: