# Impact of sample size on metric lift

Assume we ran three randomized user split AB experiments exp_1, exp_2, exp_3 one after the other. All three had the exact same treatments & control.

Exp_1 was ran for 2 weeks during 06/01-06/14 . Exp_2 was ran for 2 weeks during 06/15-06/28 and Exp_3 was ran for 2 weeks during 06/29-07/12.

Exp_1 had 2% of user population participated in the experiment, Exp_2 had 50% of the user population in the experiment, and Exp_3 had 100% of the user population in the experiment.

95% CI for primary metric lift for Exp_1 was 2.3% - 13.0%

95% CI for primary metric lift for Exp_2 was 6.2% - 8.4%

95% CI for primary metric lift for Exp_3 was 5.9% - 7.4%

From these results can we conclude that all three experiments had same metric lift? If yes/no, why ?

I was expecting the metric lifts across these experiments to be the same. Because the treatments are the same for three experiments. Only difference among them is the number of users exposed to the experiment. So, what I wanted to understand is will the metric lift change depending on the number of users exposed to the experiment. If it does, then we will have to always run experiments with 100% users exposed. This is not ideal, as we can't run other experiments at the same time.

I assume that model assumptions for the CIs are fulfilled, which in reality may not be the case.

Still the answer is no. For starters, your CIs are 95% CIs and they may not cover the true metric lift (ML; I don't know what that is by the way but chances are it doesn't matter). Note in particular that the probability that three 95% CIs all contain the true value is lower than 95% (multiple testing), and if you want a 95% coverage probability for all three CIs simultaneously, you may need Bonferroni adjustment, i.e., CIs at 1-0.05/3. There are better tests for the null hypothesis that all three MLs are the same than computing three CIs adjusted by Bonferroni and looking whether the intersection is non-empty. (I assume that this is the logic you applied here to say all three MLs might be the same, which is really not the best way of assessing this.)

Even leaving aside this issue, not rejecting a null hypothesis by a test does never mean that the null hypothesis is true (apart from pathological counterexamples that are irrelevant in practice). Rather obviously, looking at your CIs, it is compatible with the data that the first ML is 3.9%, the second is 7.5% and the third is 6.0%, so they may well not be equal (and actually it would be rather astonishing if they were exactly equal).

(1) The OP seems to have background knowledge that makes it realistic that MLs are in fact exactly equal, in which case this is of course not quite as astonishing as I initially thought. (One may think of a Bayesian analysis with a certain not too small prior probability that MLs are the same.)

(2) Although no statistical analysis can make sure that MLs are exactly the same, equivalence testing could exclude the possibility that they are very different with a low error probability. This would rely on the researcher specifying a tolerance for MLs being "similar enough" to be treated as approximately equal.

• Could the person who voted this down please tell us what's wrong with it? Commented Jul 9 at 12:32
• op is inexperienced, so rather focus on what they should do, eg equivalence test? Commented Jul 9 at 13:39
• @seanv507 It's a Q&A site. The OP has asked a question and I have answered it. As long as there's nothing wrong with the answer, I don't see why one would downvote it. If it is in fact wrong I'd like to learn why. Of course it may make sense to tell them what they should do beyond just answering their question, and you are welcome to do so writing your own answer. Commented Jul 9 at 14:37
• Thank you for the response. I was expecting the metric lifts across these experiments to be the same. Because the treatments are the same for three experiments. Only difference among them is the number of users exposed to the experiment. So, what I wanted to understand is will the metric lift change depending on the number of users exposed to the experiment. If it does, then we will have to always run experiments with 100% users exposed. This is not ideal, as we can't run other experiments at the same time. Commented Jul 9 at 15:46
• OP, please add this clarification to the question Commented Jul 9 at 16:18

OP as you have stated - the purpose of your analysis is to see whether the results do not change depending on the number of people in the test. (and this is related to your previous question Impact of user allocation on AB experiment results)

As mentioned in the previous question, unless there are specifics of your setup that you haven't mentioned, then taking a random sample of users should give a similar result as performing an ab test on all the users - it should be unbiased but have more variance.

As you mentioned, there are likely to be seasonality effects that you have not included, which are likely to affect the tests.

@dimitriy suggested one possible issue is 'interference effects' (see eg Detecting Interference in A/B Testing with Increasing Allocation. testing a change on one group of users in a social network may have effects on friends of those users (who were not included in the test). Dropping the price of one brand of jeans may cause sales of other brands to drop. Changing the colour of the web page for certain products is unlikely to have knock on effects on other product sales.

You have to decide whether this is likely in your particular case - some changes will have interference effects/some will not - so you cannot conclude that there will be no effect of user allocation with a single change (experiment).

As @christianhennig mentioned, you can't prove the null hypothesis that there is no difference, or that they are exactly the same. What you can do is an equivalence test, where you decide on minimum relevant effect size. eg +1%, below which you treat the differences as immaterial.

see the TOST procedure in the wiki page article on equivalence tests So say you take a minimum effect size of 1%, and the difference between means of exp_3 and exp_2 is -0.65%. So first assuming the null hypothesis that the true difference is +1%, you do a one sided t-test on your actual difference ((-0.65% - 1%)/stddiff), testing whether the difference is significantly below +1%. Then you do the same for the other side. Now assume that that the true difference is -1% testing if the difference is significantly above -1%: (-0.65% - (-1%))/stddiff). If the two tests are significant, then you have shown that the difference between exp_3 and exp_2 is within +/-1%.

You haven't specified how you are calculating lift, and the purpose of the metric. what worries me is eg calculating percentage increase in sales per user. eg if many low purchasing users have a higher lift, then you might conclude wrongly that total sales would increase hugely after your experiment, because the mean lift per user increased.

Also, there is often little reason to avoid running multiple tests at the same time. Unexpected interactions are typically rare, and so it is rather better to run multiple experiments in parallel than to create a bottleneck (controlled experiments at scale. You might also benefit from his book, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing http://robotics.stanford.edu/~ronnyk/

I would recommend you hire a statistician to set up these analyses for you.

• I edited out the term "significant" in "you can't prove the null hypothesis that there is no significant difference", as the term "significant" doesn't refer to the underlying parameters (hypotheses) but rather to a test result computed from data. Otherwise great effort taking into account further information on what the OP may need. Commented Jul 10 at 8:59
• Thank you for the response. I set up these experiments after reading the paper you linked . I'm concluding that I don't have enough evidence to reject the null hypothesis that lifts across experiments are different. I ended up calculating test statistic for difference in lifts. For eg: (lift_1 - lift_2)/sqrt(se_lift_1^2 + se_lift_2^2). If this is less than 1.96 (for alpha=.05) , then I can't reject the null hypothesis that lift_1=lift_2. I repeated the same for lift_1&lift_3, lift_2&lift_3. Commented Jul 11 at 19:33