The famous Monty Hall problem and what it has to do with A/B testing ...
Adam Rifkin stashed this in A/B Testing!
This is the situation: you are a contestant on a game show and the host offers you three doors to choose from. He informs you that behind two doors are cheap items and behind one is the real prize.
You choose a door, say, door A, and instead of opening this door to reveal its contents, the host opens one of the two doors you didn’t pick, say, door B. Important detail: he always picks a door with a cheap item. He then asks, do you want to stick to your choice or switch to the other closed door, door C.
What usually follows are sighs of doubt, thinking, looking for cues from the host and finally the answer, yes, I would like to change, or, no, I will not change to the other door.
What is the wise move? Should you switch doors or not? Many people say ‘how would that help? Probability stays as it was: 1/3’ or ‘now we have two closed doors each with a probability of ½, but changing doesn’t help anything’.
Those people are WRONG.
If you don't have the patience to read the whole article, I can summarize it for you:
Make sure your sample size is large enough so that it is possible to get a statistically significant result.
Sample size matters.