0
$\begingroup$

Will the ab test results change as the the number of exposed users change?

Assuming , we have the ability to expose a certain % users to an AB experiment. For eg: we can specify that only 20% of the overall user population should be exposed to an experiment. Also, users in one experiment will not participate in any other experiment.

In this case, let's say we got a statistically significant metric lift (for eg: revenue) of 5% for a 10% user allocation experiment (only 10% of the population was exposed to the experiment). Would we have got the same 5% lift if we had ran the same experiment with a higher user allocation (20%, 50%, 100%, etc..) ?

I set up some tests to answer the question above and I got results to suggest that as the user allocation changes, metric lift also changes. Doe this make sense? Any comments/thoughts will be super helpful. Thanks!

Improve this question
$\endgroup$
6
  • $\begingroup$ assuming the users are selected randomly, then the lift should "stay the same", the variation would be expected to be random. have you got eg confidence intervals on your results. $\endgroup$
    – seanv507
    Commented Jun 20 at 18:34
  • $\begingroup$ Yes random allocation. I did 3 experiments in parallel. All 3 have same treatment. Only difference is user allocation. Revenue lifts are . all statsig: 2% user allocation: +10.8%. 35% user allocation: +7.7%. 63% user allocation: +8.4% $\endgroup$
    – bp0308
    Commented Jun 20 at 20:49
  • $\begingroup$ stat significance basically means your confidence interval doesn't overlap zero. eg 10.8% +/- 5%, 7.7% +/- 1.5% 8.4% +/-1% is consistent with what you have reported). Your confidence intervals get narrower as you have more data (with square root of number - if you double the samples, your confidence interval is reduced by a factor of (1/sqrt(2)). $\endgroup$
    – seanv507
    Commented Jun 21 at 8:12
  • $\begingroup$ So can you extract the actual confidence intervals of your calculation, and that can resolve if its random noise or there is an issue in your ab tests. $\endgroup$
    – seanv507
    Commented Jun 21 at 8:14
  • $\begingroup$ Is interference possible here? $\endgroup$
    – dimitriy
    Commented Jun 21 at 8:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.