Lessons Learned

This page describes the main lessons learned from working with a wide variety of users, and how they have been driving the Magellan project. This page is still under construction.

Scale Really Matters

Many real users that we have worked with routinely must match tables of hundreds of millions of tuples, often at regular intervals (e.g., weekly or monthly). 500M tuples is not uncommon, and we have seen several cases of 1B+ tuples. They want blocking and matching tools that scale—at the very least, tools that can be applied to these tables and will run to completion. The reasoning is that if the tools cannot even complete in a reasonable amount of time, then accuracy is irrelevant.