0
$\begingroup$

I'm trying to understand 2 things in relation to calculating uplift from a marketing email campaign:

  1. Impact of statistical significance
  2. Impact of highly skewed data on calculating uplift

The example is based on the example used in this blog (#2).

EMAIL CAMPAIGN XYZ  TEST GROUP  CONTROL GROUP
Send Size                9500        500
Customers Purchased      2300        100
Conversion Rate           24%        20%
Average Purchase          250        180
Sales Uplift            575,000      18,000 

To work out the total amount that my target group (population) would have spend if we did not run the campaign, I would do:
10,000 x 20% x 180 = $360,000

To work out the uplift I would do:
(575,000 + 18,000) - 360,000 = $233,000

So my questions are:

  1. How does is the above calculation effected by statistical significance?
    For example, if I find that there is no statistical significance between 24% and 20% (hypothetically). Does that mean that doing this calculation is no longer relevant since the campaign did not have a significant impact on the proportions?
    Similarly, what if there is no statistical difference between the Average purchase amount?
    What about if there is a significant difference in proportion but not significant difference in average purchase?

  2. Let's say that the data is highly skewed and does not adhere to assumptions for two-sample t-test (purchase amount). I would then use a non-parametric test to look for statistical significance. Let say I find that there is a statistical significance.
    Is there a way to use median instead of mean for the above calculation?
    Or is it still ok to use the mean purchase amount (even though it's skewed) because we are trying to get to the total spend of the entire population without the campaign?
    In other words, it doesn't matter that the data is highly skewed, and that the mean is very different from the median. We want to know the total dollar uplift, not the middle value of spend. Therefore, we can use the average figure.

$\endgroup$

1 Answer 1

1
$\begingroup$

If you conclude that there is no difference between a pair of numbers (e.g., 20% and 24%), then yes, you should assume they are the same when doing any calculations.

If you are performing statistical tests, you've got three comparisons: 20-24, 250-180 and 60-34 (i.e., the product of the value and the percentage). The traditional testing strategy (General to Specific) would be to first compare 60-34, and then, if that difference is significant, compare 20-24 and 250-180 to work out if the difference is driven by the conversion or the value. This does solve the issue you are worried about but is a bit of a hack, and the core problem here is surprisingly complicated and gets into philosophy (frequentists versus likelihood versus Bayesian).

The mean is almost always more informative than the median. People test medians when tests of means aren't robust enough to be relied upon. If your sample sizes are accurate, then the t-test assumptions regarding the skewing aren't going to be much of an issue. It's only with small samples that you typically need to worry about such things. But, if you have a few outliers that would cause a difference. I'd suggest both testing based on median and mean, and if they give you the same conclusion then you've got no problem. If you find their conclusions are different, you need to work out why by looking at the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.