"By the end of the (Obama) campaign, it was estimated that a full 4 million of the 13 million addresses in the campaign’s email list, and some $75 million in money raised, resulted from Siroker’s careful experiments."

My favorite part is the end of the article:

Yes, Google has built its empire by listening to data, but we reserve our awe for the sort of vision that Steve Jobs brought to Apple, and we nod along at the famous answer he gave when asked how much market testing he did for the iPad: “None,” he said, echoing Henry Ford. “It’s not the consumers’ job to know what they want.”

And in fact, it’s impossible to imagine how to arrive at something like the original Macintosh, with its lack of expansion slots and its impregnable chassis, entirely through evolutionary tweaks. How could the no-slots version possibly have won over the slots version? How could a one-button mouse edge out a two-button mouse? Yet somehow a number of ostensibly negative features, when combined in a precise way, result in something serene, elegant, and Zen.

It’s a false dichotomy, of course, to pose vision against data, lofty genius against head-down experimentation, as if companies are forced to choose between the two. Every firm ought to test the small stuff, at least; and no firm should (or does) use A/B for everything. Google doesn’t test things at random but relies on intuition and, yes, vision to narrow down the infinite number of possible changes to a finite group of testable candidates.

Exactly. It's inherently reductionist.

I also read an excellent piece a few months back (sadly cannot find it now) about how A/B testing is inherently amoral and, if you're not careful, can lead to dubious methods. Since all you're concerned with is conversion rate you could end up (inadvertently) doing things misleading to users that nonetheless get them to click at higher rates.

If you ever do find it, please post it.

I've created an A/B testing stash because I want to learn more about the subject.