One time we met an associate at a VC firm who told us we should be A/B testing everything all the time. Based entirely on this, Panda and I knew right away that she'd never spent one single day working in a startup... we even correctly guessed the product at Google that she had worked on (boring but "successful"). :)

A COMPANY CANNOT TEST QUALITY INTO A PRODUCT!

Oh I see, the Jamie Quint article backs up your point:

A/B tests are very very expensive for most startups at the time when they matter most, early in their formation.

If you have to A/B test a feature then you're in Google, not a startup.

If you're a startup, you better NOT be doing A/B tests.

That said, Mike Abbott seems to have a soft spot in his heart for Optimizely:

http://techcrunch.com/2013/11/13/founder-stories-optimizely-pete-koomen/

Or maybe he's just fond of their ability to survive:

For this discussion, I sat down with with Optimizely co-founder Pete Koomen to discuss how he and his cofounder Dan Siroker left Google and started to build companies together (after Siroker’s stint with the Obama campaign in 2008 and thereafter). The duo launched an educational site to teach kids math, but felt disconnected from the target users and weren’t confident the service could grow. Later, they got into Y Combinator and, during that incubation period, dropped their initial idea (Company #2) and pivoted into what would become Optimizely. The rest, as they say, is history — for now.

What struck me most about this conversation, as it was unfolding and as I’m watching it again, is that if you listen closely, you can hear the desperation in Koomen’s voice when he talks about tasting failure during the first two companies. You can hear the stress about not having a paycheck for a year. You can hear the ambition of someone who wanted to do something on his own, and was struggling to find that one “thing.” There seem to be some tough lessons in this story. Koomen and Siroker were accomplished technical engineers, held great positions in and out of Google, and had some of the best angels in the Valley behind them — and struggled for a while until they got to a point where they were revenue-positive and could focus on building the type of company they wanted to.

Three funny points about this story:

1. Optimizely is an A/B Testing company founded by Googlers because they didn't believe in an educational site to teach kids math but they do believe in A/B Testing the crap outta things.

2. Abbott uses the phrase "the rest as they say is history -- for now" meaning he too is skeptical that an A/B Testing company might not have a long term big future ahead of it.

3. By forcing its companies to be revenue-positive YCombinator stopped many of its companies from doing good things that could make the world better (Teach Kids Math!) to instead work on boring enterprise companies (Testing Framework SAAS!). Ugh.