Google Made Its Secret Blueprint Public to Boost Its Cloud
J Thoendell stashed this in Tech
Over the past decade, the company would build software for running code or juggling data across hundreds of its own machines. It would jealously hide this software, seeing it as a competitive advantage. Then, a few years down the road, it would release a research paper describing the tech, and the open source world would clone it. This happened with so many tools, including MapReduce (which spawned the open source Hadoop) and BigTable (which gave us a world of “NoSQL” databases).
Much the same thing happened with Borg. A few years ago, engineers at Twitter and the University of California, Berkeley built a tool called Mesos, and today, it underpins some big-name internet services, including Twitter and Airbnb. And several other projects are building similar tools around a technology called Docker, which helps developers neatly package their applications into the sort of software “containers” that Borg juggles across the Google network.
But now, driven by the needs of its cloud business and other forces, Google is changing its role. It’s not just sharing a research paper. It’s not just open sourcing small pieces of its online infrastructure. It’s transforming a major system into an open source project—at least up to a point.
As Burns puts it, he and Beda and McLuckie saw so many other running projects that explored the basic ideas behind Borg and containers, and they decided Google could help push things forward. “We really felt like all these puzzle pieces were out there and people were stumbling around trying to put them together,” he says. “We really had the puzzle box. We knew, from ten years of experience, how the pieces all fit together.”
To be sure, Kubernetes is not an open source Borg. It’s not as complex as Borg or Borg’s successor, known as Omega. And as it stands, it won’t juggle tasks across quite as many machines. “We’re still reaching the same scale as Borg,” Beda says. “The focus for Kubernetes out of the gate was getting the core concepts right.”