# Impact of user allocation on AB experiment results

Will the ab test results change as the the number of exposed users change?

Assuming , we have the ability to expose a certain % users to an AB experiment. For eg: we can specify that only 20% of the overall user population should be exposed to an experiment. Also, users in one experiment will not participate in any other experiment.

In this case, let's say we got a statistically significant metric lift (for eg: revenue) of 5% for a 10% user allocation experiment (only 10% of the population was exposed to the experiment). Would we have got the same 5% lift if we had ran the same experiment with a higher user allocation (20%, 50%, 100%, etc..) ?

I set up some tests to answer the question above and I got results to suggest that as the user allocation changes, metric lift also changes. Doe this make sense? Any comments/thoughts will be super helpful. Thanks!

• assuming the users are selected randomly, then the lift should "stay the same", the variation would be expected to be random. have you got eg confidence intervals on your results. Commented Jun 20 at 18:34
• Yes random allocation. I did 3 experiments in parallel. All 3 have same treatment. Only difference is user allocation. Revenue lifts are . all statsig: 2% user allocation: +10.8%. 35% user allocation: +7.7%. 63% user allocation: +8.4% Commented Jun 20 at 20:49
• stat significance basically means your confidence interval doesn't overlap zero. eg 10.8% +/- 5%, 7.7% +/- 1.5% 8.4% +/-1% is consistent with what you have reported). Your confidence intervals get narrower as you have more data (with square root of number - if you double the samples, your confidence interval is reduced by a factor of (1/sqrt(2)). Commented Jun 21 at 8:12
• So can you extract the actual confidence intervals of your calculation, and that can resolve if its random noise or there is an issue in your ab tests. Commented Jun 21 at 8:14
• Is interference possible here? Commented Jun 21 at 8:58