6
$\begingroup$

Before starting an a/b test with a control and one experimental route, I can calculate a required sample size based on conversion rate estimates for both routes. I can get a good estimate of the conversion rate of the control by looking at historical data. But the conversion rate of the experimental route is unknown. What I would like to do is calculate a number of different sample sizes based on a variety of sensitivities.

For example, I can calculate sample sizes for a 10%, 15% and 20% sensitivity (increase in conversion from the control) that might look like this:

Sensitivity   Required Sample Size
10%           1,961
15%           871
20%           490

Some of the reading I've done says that you should calculate a single sample size at the start of the test and always run the test for that long.

Question:

  • Is there any problem with checking for statistical significance at multiple pre-calculated sample sizes and potentially ending a test early if I've found the results to be statistically significant?

Example:

I originally estimate that the experimental route will outperform the control by 15%. But once I've reached 490 samples I find that the experimental route is actually outperforming the control by 20%, can I end the test and declare that the experimental route boosts conversion by 20%?

Improve this question
$\endgroup$
6
  • $\begingroup$ You can stop your test at any point (nothing wrong with that), but in this case your result is still 20+-20%, taking into account your sensitivity. So even the probability that there is no effect is high. I strongly suggest this read: visualwebsiteoptimizer.com/split-testing-blog/… and that blog in general. $\endgroup$
    – sashkello
    Apr 22, 2013 at 22:58
  • $\begingroup$ @sashkello: Are you saying that a test stopped at 490 samples because one subject is shown to be outperforming the other by 20% somehow reports a value that is less credible than one that collected 1,961 samples and also showed the same 20% performance improvement? $\endgroup$
    – jkndrkn
    Apr 23, 2013 at 18:07
  • $\begingroup$ @jkndrkn Of course, they will have different precision. What if you have test of 5 samples with 20% (one sample) improvement? It has no credibility whatsoever. $\endgroup$
    – sashkello
    Apr 23, 2013 at 22:50
  • $\begingroup$ @sashkello: Assume the significance level of the experiment is 0.95 and the power is 0.8. Say the user of the test is comfortable stopping at 490 samples because the effect is shown to be 0.2. The user knows that with fewer samples the credibility of the test is reduced. Can a credibility score of some kind be assigned that accompanies a sample size? If so, which value could be used? $\endgroup$
    – jkndrkn
    Apr 25, 2013 at 19:59
  • 1
    $\begingroup$ @sashkello the sample size for 20% sensitivity was calculated using a significance level of 95% and a power of 80%. So if a 20% effect is observed at 490 samples, can the test be ended and considered credible? $\endgroup$
    – nates
    Apr 25, 2013 at 23:01

1 Answer 1

Reset to default
2
$\begingroup$

This approach doesn't have the properties you would have if you fixed the sample size ahead of time.

The situation where you look for a particular result while your experiment continues and have some 'stopping rule' (halt your experiment early if a particular situation is achieved) is a version of sequential analysis; see also SPRT.

You have to take care that the properties of your actual decision rules are doing what you want - you can't apply the properties of one situation to another and expect that it will work.

For example, you won't have the power you have calculated at the given sample sizes if you're doing sequential testing; the required sample sizes will be somewhat larger. On the other hand, when your effects are substantial, you'll often end up stopping earlier - meaning smaller sample sizes/faster decisions.

Specifically which properties are affected if one were to terminate the test at, say, 490 samples because 20% improvement over the control is shown?

First, estimates will be biased, but also standard errors, Type I and (as already mentioned) Type II error rates are affected - plus anything any of these will impact.

The SPRT link I gave outlines a general approach that is used with early stopping with hypothesis testing.

Phillip Good does some work with discrete sequential analysis in his book Permutation, Parametric, and Bootstrap Tests of Hypotheses in section 6.7

Improve this answer
$\endgroup$
4
  • $\begingroup$ Specifically which properties are affected if one were to terminate the test at, say, 490 samples because 20% improvement over the control is shown? $\endgroup$
    – jkndrkn
    Apr 23, 2013 at 18:08
  • $\begingroup$ See the edit for some details of the kinds of things affected. $\endgroup$
    – Glen_b
    Apr 24, 2013 at 8:58
  • $\begingroup$ Thanks, is there any way to quantify how bias, Type I, and Type II error rates are affected? It may be desirable to end a test early so long as these parameters don't suffer beyond some acceptable threshold. $\endgroup$
    – jkndrkn
    Apr 25, 2013 at 20:44
  • $\begingroup$ @jkndrkn (My apologies for not @-notifying you before, I hadn't realized you weren't the OP until just now. ) ... Yes, you can quantify the bias under a given set of assumptions via simulation, for example. But why would you choose to do something biased instead of actually using a proper sequential analysis in the first place? $\endgroup$
    – Glen_b
    Apr 25, 2013 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.