We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, and within those that do, it is heavily skewed towards lots of small spenders and a few very big spenders), so we've decided on using bootstrapping to compare the means, to get round the issue of the data not being normally distributed.

So my results show that we do have an uplift of around 8% vs. control. I now want to calculate how confident I can be in this uplift. Is it as simple as measuring the proportion of the probability density function below zero, for the PDF that is test group PDF minus control PDF? (e.g., that portion reflects the % chance that my 2 PDFs are not different?)

Any help would be much appreciated.

  • $\begingroup$ What is A/B test? $\endgroup$ – StasK Oct 9 '13 at 13:55
  • $\begingroup$ a one-factor experiment: eg we split our users equally in to two randomly assigned groups. One was shown the new feature and the control group just saw the normal vanilla product without the new feature. We then compared the target metric (in this case revenue) for the test group against the control group to see if those with new feature spent more (or less) than the control group $\endgroup$ – user31228 Oct 9 '13 at 14:53

Bootstrapping for a mean does not usually make sense for due to the CLT. Just use the mean and standard error on the mean. This will either give the same result as your bootstrap or your bootstrap would have given a poor result.

If you truly know you want to compare the mean (and it is not clear that you do Better estimator of expected sum than mean) then you want to test if the two samples are likely to have come from the same population. This would be a Welch t-test with the mean, $\mu$, and standard error on the mean, $\sigma_\mu$.

The problem becomes more complicated if there is a different associated risk with each choice coming from other factors (eg implementation cost). If the t-test says there is no significant difference then you clearly want to take the variant with lower risk. However, if the variant with higher risk has a statistically significant impact on mean revenue then it is a judgment call.

| cite | improve this answer | |
  • $\begingroup$ Linking to one of your own questions (which has at present not been found clear enough to prompt any anwers yet) is not helpful. $\endgroup$ – conjugateprior Aug 18 '14 at 10:11
  • $\begingroup$ The lack of answer is likely because the problem is intractable. As explained in the third paragraph there are complications with the problem that the questioner may not have considered. The question I asked highlights the problem associated with the assumption that the mean is the best estimator. $\endgroup$ – Keith Aug 18 '14 at 10:18
  • $\begingroup$ Perhaps so, but the reasons for the lack of answer there are not relevant to whether it is helpful to link to it here. The only reason to link to one of your own questions elsewhere would be because it contained an answer that the OP here might use. $\endgroup$ – conjugateprior Aug 18 '14 at 10:46
  • $\begingroup$ It contains relevant information for OP to consider. Either way, the question is answered after 10 months and you are complaining that I gave too much information. If you are confident OP understands why he chose the mean then you are welcome to edit my answer. $\endgroup$ – Keith Aug 18 '14 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.