Discussion inspired by the Ghost in the Pixel blog post Why designers don’t like A/B testing by Uday Gajendar.

The blog post was written in response to this tweet: http://twitter.com/#!/lauraklein/status/73135201416527873

I don't trust designers who don't want their designs a/b tested. They're not interested in knowing if they were wrong. #leanstartup

Gajendar proceeds to explain why it's basically impossible for designers to be wrong. Though he makes a couple of good points, I find myself unable to ignore the smell of designer arrogance, wafting up from the page in waves like the fumes of a beached whale decomposing on the beaches of Santa Cruz, California.

Probably the most irritating of Gajendar's claims is the following:

Users are not always the best judge of high quality design. That’s why you hire expert designers of seasoned skills, experience, judgment, and yes the conviction to make a call as to what’s better overall.

O RLY??? Are there any objective metrics by which a designer may be assessed, then? Or must I get a master's degree in HCI before I'm qualified to discuss possible weakness in your design (such as the fact that it's too complicated for users to understand)?

While I don't believe in A/B testing any more than Gajendar (at least not for startups), it really seems to me like his attitude can be summarized as follows: The designer is right. If the users don't get it, they're stupid and lack training in UI/UX. If your A/B testing suggests another design might be better, than you're a n00b for using A/B testing instead of listening to me -- the expert.

It seems that so many designers feel like they are accountable only to their own sense of Platonic design ideals, rather than to, say, the company and its mission. They often emphasize the importance of having a grand unified vision for the sites design, and breaking features down into conceptual categories that are probably more meaningful to the designer himself than to the average user.

And when it comes down to it, aren't you building the product for your users? Doesn't their opinion matter quite a lot? Granted, you don't succeed by listening to everything users say, but it's nonetheless crucial that you incorporate direct and indirect user feedback in all its forms: entries in the error logs, use (and misuse) of features, emails to support staff, etc.

Look at Quora as a case study. Listening to people "in the know" (i.e., UI/UX and designers in Silicon Valley) Quora is "like Yahoo! Answers, but better because it focuses more on UX." But when you let designers run wild, completely unaccountable to any kind of objective reality, the result is a poorly socialized product which does not play nicely with the average user.

I don't mean to knock either designers or Quora, but I do think that, just like everyone else, designers should remember that they are a member of a team, and that they are really there to help the team figure out how to make the product successful, not create a work of art.

Yes, we build products for our users.

A product is a living, growing, evolving thing. It starts with a vision, and then it evolves once its creators see how people are actually using it.

In practice, we never know how people will use a product until people actually use the product.

Despite the school of What would Steve Jobs do?, it's the users who make the product successful, not the designers.

See also: What is the ideal relationship between design and engineering in a startup?

It is the users who ultimately make the product useful, and it's why whoever's listening to what your users are saying has to be not just a good listener, but someone you trust to be a competent filter in order to be able to make sure the product being built is still one the users want.

Just my two cents, and that's probably all my feedback's worth :P

Isn't this a little bit like saying is doctor better judge of the illness the patient has instead of the patient who is actually ill.

You could look at it that way if you feel that designers know something that the rest of us do not.

I personally feel like it's similar to the a musician saying that they are a better judge of music quality than people who simply enjoy listening to music.

If most people agree that some musician's latest album is not good, do we necessarily trust that less than the musician's opinion (which is probably that the unwashed masses lack the refined palate to appreciate "truly good" music)?

But I'm personally of the opinion that UI/UX design is more akin to art than to science.

Evidence-based design is BOTH an art AND a science: http://www.uxmag.com/design/the-art-and-science-of-evidence-based-design

I prefer the words of Patrick Kennedy, who said, "The scientist and artist within all of us should be grappling with each other to keep each on the level."

Source: http://www.gurtle.com/ppov/2009/06/03/is-ux-an-art-or-a-science

I am ancient by all accounts - having studied UI/UX design in the *80's* (!?) at Cal, but I did happen upon an interesting insight (I think) while doing my 3rd start-up - an education company. After sitting through endless presentations on good learning design (blah blah blah "different learners = different learning styles" blah blah blah), I came to the belief that good UI/UX design shares a lot in common with good learning design: different users, different styles (better listen to A/B feedback at some point, lest you miss a constituency you don't understand); inspirational educators innovate in ways that are so far out-of-the-box that it is difficult to evaluate the technique until it is (nearly) fully formed and in use (supports the "god-designer" belief of no A/B testing); all good learning design incorporates continuous feedback (A/B testing); the only "complete" education material is on a topic that is no longer relevant... All UI Designers are at heart educators of software use patterns...

Feedback loops are powerful things, that can be employed both to create and to destroy:

"A feedback loop involves four distinct stages. First comes the data: A behavior must be measured, captured, and stored. This is the evidence stage. Second, the information must be relayed to the individual, not in the raw-data form in which it was captured but in a context that makes it emotionally resonant. This is the relevance stage. But even compelling information is useless if we don’t know what to make of it, so we need a third stage: consequence. The information must illuminate one or more paths ahead. And finally, the fourth stage: action. There must be a clear moment when the individual can recalibrate a behavior, make a choice, and act. Then that action is measured, and the feedback loop can run once more, every action stimulating new behaviors that inch us closer to our goals."

Source: http://www.wired.com/magazine/2011/06/ff_feedbackloop/all/1

@ifindkarma Actually, Adam, your comment hit the nail on the head. There is art in the balance. Designers dream. Users (in)validate.

It is the dreamers who create. No wonder it's so hard.

 I sent this to Uday, so he can comment as well. BTW, I think the Pandawhale email is missing out sending me the best conversations... I was sad to have missed this one. 

AB testing is an optimization strategy. People keep forgetting that. You can AB test your way to innovation.  AB testing is great for fine tuning something that is essentially solved. Solving is harder. 

I love AB testing. Love Love Love it. But if I threw up crap into an AB test, and tried to get a great design (great=converts) I'd be at it a long time. Good AB testing (or multivariate) is slow, and doesn't work unless you have enough users anyhow. Better to work with great product and design folks, and have them make something that is as terrific as we can imagine, define core questions, and set them into testing (qual or quant) to get at answers. 

AB testing is annoying because so many business folks think it's a hammer that can put screws into wood. if you follow my mangled metaphor. With similar interface results http://static2.bigstockphoto.com/thumbs/2/7/2/large2/2723058.jpg 

Well said: "Design to begin: A/B testing to optimize."

Thanks Christina for passing this along! It's always fun to see other's take on my piece written over a year ago...

Ironically (or coincidentally) I learned of this discussion right after hitting "send" for an email to my boss (the SVP of Customer Experience) requesting her to please give me user research resources to run studies on our latest designs :-) I think that alone validates (ha!) my longstanding view of research and it's relationship to enabling "good" designs that shape a compelling vision, yet address optimal needs. It's a balance of tools and approaches. No one right way.

Criticizing A/B testing does not mean one is against gathering user feedback. That's just silly!

It's important to note, that when you hire a senior (7-10+ yrs) designer like myself, just like when you "hire a doctor" or "hire a tax attorney", you (as product team) are placing part of your trust in their education, training, practice, and cultivated expertise and judgment that's evolved over the years. Having degrees in HCI and MDes arm the designer with baseline rationale for articulation. Patterns are detected, principles ascertained, and practices applied. It's not about "the design god" or "ego-driven design", which says more about the person who holds that opinion about designer's role, than the designer itself. It's about recognizing wizened craft informed by understanding people over time (aided by studies along the way) that might just perhaps leap ahead of what a user thinks they know, yielding a product or service that evokes gratitude and delight. 

 I'll always remember a brilliant designer who worked for me placing Atul Gawande's "Whose Body Is It Anyway?" https://mywebspace.wisc.edu/rstreiffer/web/CourseFolders/MHB558S06/Gawande%20-%20Whose%20Body%20Is%20it%20Anyway.pdf on my desk and saying "replace body with design and doctor with designers"

Product always lives with consequences of design decisions the way designers never have to, yet design holds the expertise (one hopes). 

Wait, why do designers not have to live with consequences of design decisions like product people have to?

 Think about how a designer and a PM are evaluated. PM is evaluated on numbers. designers are usually evaluated on how well they collaborate and how well their designs are received internally. Like engineering, they are usually a support team, and they get judged on how well the support. If they fight hard for a design that lowers the conversation rate, and the PM ok's it, the PM is the one who lives with the results.  Just as the doctor says, try this pill or do surgery, but the patient lives with the results. The doctor does his/her best, and if they make a lot of bad decisions they may have less patronage, but overall they are.. in agile terms.. chickens, not pigs.