0
$\begingroup$

If I run an A/B with the recommended sample size (using this for example), and at the end my results are not significant, what can be done at this point? If someone requests to continue running the test (i.e. increase the sample size), what implications does this have?

I guess this will cause a "peeking" problem, but how do you quantify this if someone asks you to, say, double the sample size? And if I do double the sample size, what is the likeihood of getting significant results, given that the recommended sample size did not produce significant results?

So a specific example

  1. I run a test with a recommended 10K impressions on each variation
  2. At the end of this test, the p-value is above 5% so there is not significant results
  3. Someone suggests to continue running the test to see if it does produce significant results. Is this a bad or good idea? And how do you motivate the answer?
$\endgroup$
  • $\begingroup$ If you're interested in what looking twice does to the significance level, you should look into methods of sequential design $\endgroup$ – Cliff AB Jan 7 '16 at 20:13
1
$\begingroup$

What is the cost of running the additional time?

What is the cost of a type I error (finding significance when there really is not a difference)?

What is the cost of a type II error (not finding a difference that is there)?

How would the decision be made if you don't run any longer?

Yes, you are looking at a case of the peeking problem, or multiple comparisons, but the consequences range from dire to irrelevant depending on the answers to the questions above.

For example, if there is really not much cost to running longer (just your time sometime in the future to run a second test) and if you don't run longer then the decision will be made by flipping a coin, then getting more data at worst is a slow coin flip and will not really hurt.

On the other hand, if running longer deters you from finding a better solution, or could lead to a more expensive solution when the cheaper is just as good, then the cost of getting more data is more severe.

One way to quantify the effect would be to simulate data. Simulate data where the null is true at the final sample size, then analyze just the first part and the whole data. Repeat this a bunch of times to see how often the 1st test is not significant and the 2nd is. Also include any known costs. You can also redo this with the null false at various differences to see those effects as well.

$\endgroup$
0
$\begingroup$

When you first chose your sample size using the calculator, you entered a "minimum detectable effect". If this level that you entered is the smallest effect that would be relevant for you (which would have probably been the best way to run the test, if not cost prohibitive), then you should stop, since it is highly likely that the true difference, if there is any, is below this minimum detectable effect.

If, however, you would care about an effect size smaller than the minimum detectable effect you entered at first, it may be worthwhile to continue. Adjusting the power and minimum detectable effect sliders on that calculator will give you a better sample size needed for your application.

As you mentioned, this does lead to a "peeking" problem, which means your p-values will be biased downward. To see this, imagine that you were looking at the results after every impression and calculating a p-value. It is essentially a statistical guarantee that you will eventually get a "significant" p-value eventually, even if there is no true effect. The more often you check, the more biased your p-values will be when you are finished. To counter this, if you indeed decide that you would like to be able to detect a smaller effect than your original test could have, you should choose a smaller value for the significance level $\alpha$ than you would otherwise, since a low p-value does not hold as much meaning when you make decisions of whether or not to continue the test based on earlier p-values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.