8
$\begingroup$

We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, and within those that do, it is heavily skewed towards lots of small spenders and a few very big spenders), so we've decided on using bootstrapping to compare the means, to get round the issue of the data not being normally distributed.

So my results show that we do have an uplift of around 8% vs. control. I now want to calculate how confident I can be in this uplift. Is it as simple as measuring the proportion of the probability density function below zero, for the PDF that is test group PDF minus control PDF? (e.g., that portion reflects the % chance that my 2 PDFs are not different?)

Any help would be much appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ What is A/B test? $\endgroup$
    – StasK
    Commented Oct 9, 2013 at 13:55
  • 1
    $\begingroup$ a one-factor experiment: eg we split our users equally in to two randomly assigned groups. One was shown the new feature and the control group just saw the normal vanilla product without the new feature. We then compared the target metric (in this case revenue) for the test group against the control group to see if those with new feature spent more (or less) than the control group $\endgroup$
    – user31228
    Commented Oct 9, 2013 at 14:53
  • $\begingroup$ "the PDF that is test group PDF minus control PDF" what do the pdf's represent? Distribution of individuals or simulation of distribution of test statistic? How do you subtract the pdf's? What did you bootstrap? The distributions of controls and test groups? Why not compute the distribution of the uplift for each simulated sample in the bootstrap? $\endgroup$
    – Sextus Empiricus
    Commented Apr 10, 2023 at 20:29
  • $\begingroup$ Have you attempted a simple log transform? Are you aware of the ANCOVA as a type of interrupted time-series analysis? Before pulling yourself out of quicksand by the proverbial bootstraps, do check if there's a rope nearby. $\endgroup$
    – AdamO
    Commented Apr 10, 2023 at 20:52

1 Answer 1

Reset to default
0
$\begingroup$

Bootstrapping for a mean does not usually make sense for due to the CLT. Just use the mean and standard error on the mean. This will either give the same result as your bootstrap or your bootstrap would have given a poor result.

If you truly know you want to compare the mean (and it is not clear that you do Better estimator of expected sum than mean) then you want to test if the two samples are likely to have come from the same population. This would be a Welch t-test with the mean, $\mu$, and standard error on the mean, $\sigma_\mu$.

The problem becomes more complicated if there is a different associated risk with each choice coming from other factors (eg implementation cost). If the t-test says there is no significant difference then you clearly want to take the variant with lower risk. However, if the variant with higher risk has a statistically significant impact on mean revenue then it is a judgment call.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Linking to one of your own questions (which has at present not been found clear enough to prompt any anwers yet) is not helpful. $\endgroup$
    – conjugateprior
    Commented Aug 18, 2014 at 10:11
  • $\begingroup$ The lack of answer is likely because the problem is intractable. As explained in the third paragraph there are complications with the problem that the questioner may not have considered. The question I asked highlights the problem associated with the assumption that the mean is the best estimator. $\endgroup$
    – Keith
    Commented Aug 18, 2014 at 10:18
  • $\begingroup$ Perhaps so, but the reasons for the lack of answer there are not relevant to whether it is helpful to link to it here. The only reason to link to one of your own questions elsewhere would be because it contained an answer that the OP here might use. $\endgroup$
    – conjugateprior
    Commented Aug 18, 2014 at 10:46
  • $\begingroup$ It contains relevant information for OP to consider. Either way, the question is answered after 10 months and you are complaining that I gave too much information. If you are confident OP understands why he chose the mean then you are welcome to edit my answer. $\endgroup$
    – Keith
    Commented Aug 18, 2014 at 12:34
  • 1
    $\begingroup$ The CLT applies to a large number of statistics because they can be formulated as a mean - that includes regression parameters, standard deviations, the median, IQR, ACF, even the whole eCDF has a CLT, etc. So I wonder what statistics you're thinking about as ideal for bootstrapping. I'm sure most would agree that it's not the statistic that determines the bootstrap as a useful estimator, but rather the complexity of the actual sampling distribution of the statistic. $\endgroup$
    – AdamO
    Commented Apr 10, 2023 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.