EONpedia

From the beginning to the present.

Each of these racks holds ordinary computers; what MapReduce contributed is that one of them failing no longer stops the computation.

Image: erwinboogertCC BY-SA 3.0file pagemodified

2004 · Mountain View, California, USA

MapReduce and the birth of big data

Share

Two Google engineers published a simple scheme for running thousands of ordinary computers as one; data stopped being a stored record and became a raw material under continuous processing.

The MapReduce paper published by Jeffrey Dean and Sanjay Ghemawat in 2004 proposed no new mathematics. What it proposed was a division of labour: split a large dataset into pieces (map), process each piece on a separate machine, and combine the results (reduce). The real innovation was its assumption of failure — with thousands of cheap machines, some will always be broken, so the logic of redistributing work was built into the system itself. Computing power thereby shifted from a single expensive supercomputer to a replaceable pile of commodity hardware.

Hadoop, the open-source counterpart, spread within a few years and the cost threshold fell: records that had previously been discarded — clicks, search queries, sensor readings, location traces — became worth keeping. The term 'big data' describes that change of attitude more than any volume: collect first, decide later what it is for. The same infrastructure soon underpinned cloud services and, after 2012, the wave of deep learning; training modern AI models rests precisely on data and distributed computation at this scale.

The other face of that accumulation is privacy. Personal data becoming a commercial asset produced regulation such as the European Union's GDPR in the 2010s and an unfinished argument about the advertising economy. That a word typed into a search box, a purchase or a walk leaves a measurable trace is now taken for granted; the technical ground for that assumption was laid in the mid-2000s.

Gallery

Image: Carl LenderCC BY 2.0

Location

Mountain View, California, USA · © OpenStreetMap

Sources