6
$\begingroup$

Background: We know from this article that ending an A/B test early due to "significant" results is a mistake.

Question: But what about when a test runs for the desired time period and shows insignificant results – is it fine to prolong it? What are the risks?

Note: It would be great with a simple mathematical example of any risks, similar to the example in that linked article.

I have only a basic knowledge of probability theory and maths, so I would appreciate an answer I can understand with that knowledge.

Attempt: My intuition is that it could be problematic, because you had an experiment with a calculated reliability (will show false positives in X% and false negatives in Y% of such experiments), but now you're effectively waiting indefinitely for the first true-positive or false-positive significance.

So I should think you get more false positives than you accounted for when setting up the original experiment. But presumably the likelihood of false positives also decreases as we get more data.

$\endgroup$
  • 4
    $\begingroup$ (+1) When the A/B test is appropriately conducted--that is, using a sequential test rather than requiring a fixed sample--then this issue doesn't come up. $\endgroup$ – whuber Mar 24 '17 at 11:58
  • $\begingroup$ @whuber: Is this what you are referring to? evanmiller.org/sequential-ab-testing.html I would still be interested to hear what the implications of extending a fixed-sample test would be. If the answer is "don't extend them for this and that reason, but use a sequential test if you want the option to extend", I'd love to hear more about that. $\endgroup$ – Henrik N Mar 24 '17 at 21:51

Browse other questions tagged or ask your own question.