# How to analyze and interpret A/B test results via Bootstrap method?

We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, and within those that do, it is heavily skewed towards lots of small spenders and a few very big spenders), so we've decided on using bootstrapping to compare the means, to get round the issue of the data not being normally distributed.

So my results show that we do have an uplift of around 8% vs. control. I now want to calculate how confident I can be in this uplift. Is it as simple as measuring the proportion of the probability density function below zero, for the PDF that is test group PDF minus control PDF? (e.g., that portion reflects the % chance that my 2 PDFs are not different?)

Any help would be much appreciated.

• What is A/B test? – StasK Oct 9 '13 at 13:55
• a one-factor experiment: eg we split our users equally in to two randomly assigned groups. One was shown the new feature and the control group just saw the normal vanilla product without the new feature. We then compared the target metric (in this case revenue) for the test group against the control group to see if those with new feature spent more (or less) than the control group – user31228 Oct 9 '13 at 14:53

If you truly know you want to compare the mean (and it is not clear that you do Better estimator of expected sum than mean) then you want to test if the two samples are likely to have come from the same population. This would be a Welch t-test with the mean, $\mu$, and standard error on the mean, $\sigma_\mu$.