The canary release process offers stability and feedback.

Google’s first canary is google.com itself, as it has over 50,000 employees. Starting Monday and through the week, Google gradually rolls the new release internally to it’s own users, monitoring hundreds of different metrics on scalability and stability. Google’s employees act as canaries – squawking about any issues that come up. If core metrics are impacted significantly, the release is rolled back entirely. However, usually fixes are put in for critical issues and the release goes to the entire Google employee base.

On the next Monday, Google will take the release that had been rolled out to Google employees and canary release externally. Google will start showing more users the new release. Google continues to monitor for scalability and stability – using the external users as the true “staging” environment. Google’s products have a large enough base that they can even do sentiment analysis for keywords like “Gmail Sucks” on twitter. If there are spikes, Google will stop or roll back the new release. But usually by the end of the week, the entire release has been rolled out, and it’s time for the new new release.