3
$\begingroup$

If I am running an A/B test in which I have two randomly assigned groups of users and I am calculating a conversion on some action, how do I then calculate the uncertainty on the result?

For example, if group A has 1000 users of which 100 took the desired action, and group B has 1000 users and 90 took the action I would end up with a conversion rate of 0.1 and 0.09 respectively but I really want to be able to state something about the uncertainty on these rates.

My naive approach would be to assume sqrt(N) as the statistical uncertainty on each number and use the propagation of error approach, differentiating the ratio, but I'm not confident that this is the correct approach.

$\endgroup$
3
$\begingroup$

One approach would to model the distribution of the ratios, and the calculate the confidence interval based on this distribution.

For example, suppose you repeat A and B many times, and you find that you can model them as independent normal variables $A$ and $B$ (whose parameters you find). The ratio will have a Cauchy distribution. Once you obtain this Cauchy distribution, you can calculate whatever you want on it.


A different, non-parametric approach would be to use bootstrapping.

Perform many iterations of the following. In each iteration, select - with replacement - 1000 random samples from A and 1000 random samples from B, and calculate the ratio for this iteration.

Build the empirical distribution of the ratio, and calculate the confidence intervals on it.

Note that bootstrapping is known to be problematic for heavy-tailed distributions. For the specific conversion rates in your question, this doesn't seem to be a problem, though.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.