Image: erwinboogertCC BY-SA 3.0file page ↗modified
2004 · Mountain View, California, USA
MapReduce and the birth of big data
Two Google engineers published a simple scheme for running thousands of ordinary computers as one; data stopped being a stored record and became a raw material under continuous processing.
The MapReduce paper published by Jeffrey Dean and Sanjay Ghemawat in 2004 proposed no new mathematics. What it proposed was a division of labour: split a large dataset into pieces (map), process each piece on a separate machine, and combine the results (reduce). The real innovation was its assumption of failure — with thousands of cheap machines, some will always be broken, so the logic of redistributing work was built into the system itself. Computing power thereby shifted from a single expensive supercomputer to a replaceable pile of commodity hardware.
Hadoop, the open-source counterpart, spread within a few years and the cost threshold fell: records that had previously been discarded — clicks, search queries, sensor readings, location traces — became worth keeping. The term 'big data' describes that change of attitude more than any volume: collect first, decide later what it is for. The same infrastructure soon underpinned cloud services and, after 2012, the wave of deep learning; training modern AI models rests precisely on data and distributed computation at this scale.
The other face of that accumulation is privacy. Personal data becoming a commercial asset produced regulation such as the European Union's GDPR in the 2010s and an unfinished argument about the advertising economy. That a word typed into a search box, a purchase or a walk leaves a measurable trace is now taken for granted; the technical ground for that assumption was laid in the mid-2000s.
Gallery
Image: Carl LenderCC BY 2.0
Location
Mountain View, California, USA · © OpenStreetMap
Sources
- MapReduce: Simplified Data Processing on Large Clusters (Dean & Ghemawat, OSDI '04) — Google Research
- The Hadoop Distributed File System — Apache Software Foundation
- Big data — Encyclopaedia Britannica — Encyclopaedia Britannica