I read this article on "how not to run an A/B test".
And I still don't understand what exactly the author's reasoning is. Can someone dumb it down for me?
I think what it might be saying is that reading the results of my split tests over time misleads me. I want to be able to understand this well enough that I can explain it to others, though.
Any help?
It's the "best two out of three" phenomenon. You know the joke:
"Let's flip for it."
"OK, go!"
"Oops, I lost. How about flipping two more times, with the winner being the best of the three total times?"
Significance testing is exactly like coin flipping (but with biased coins, usually). If you run a short test and it's not significant, maybe you can achieve significance (partly through luck) by prolonging the testing.
The converse of this (I'm tempted to say the "flip side" of this :-)) is that if you plan to conduct a certain number of tests and happen to see a "significant" result early, that's also not dispositive. It's analogous to the reverse of our first contest:
"Let's flip for it. Best two out of three?"
"OK, go!"
"Ha, I won the first flip, so I win!"
Having said that, note that there are versions of testing which allow you to monitor the (nominal) significance as you go along. These work like ending a contest early when it gets too one-sided, so-called mercy rules. If, in the early going, it becomes extremely obvious that a difference is real, you can save time and effort by ending the testing. These are called sequential hypothesis testing procedures. A good case could be made that these should be your standard way of conducting A-B tests, because in the long run you will spend less time and effort overall.
