What happens when your userbase starts doubling EVERY DAY? Your systems guys don't get a lot of sleep, but they'll be happier if they kept things simple.

50 million downloads in 50 days.

At peak 3000 drawings per second.

Damn.

"At one point our growth was so huge that our players -- millions of them -- were doubling every day. It's actually hard to wrap your head around the fact that if your usage doubles every day, that probably means your servers have to double every day too. Thankfully our systems were pretty automated, and we were bringing up tons of servers constantly. Eventually we were able to overshoot and catch up with growth by placing one order of around 100 servers. Even with this problem solved, we noticed bottlenecks elsewhere.

"This had us on our toes and working 24 hours a day. I think at one point we were up for around 60-plus hours straight, never leaving the computer. We had to scale out web servers using DNS load balancing, we had to get multiple HAProxies, break tables off MySQL to their own databases, transparently shard tables, and more. This was all being done on demand, live, and usually in the middle of the night."

More about how freaking huge mobile is right now: http://pandawhale.com/convo/1106/instagram-and-path-and-branchout-and-evernote-and-square-oh-my