I'm trying to understand 2 things in relation to calculating uplift from a marketing email campaign:
- Impact of statistical significance
- Impact of highly skewed data on calculating uplift
The example is based on the example used in this blog (#2).
EMAIL CAMPAIGN XYZ TEST GROUP CONTROL GROUP
Send Size 9500 500
Customers Purchased 2300 100
Conversion Rate 24% 20%
Average Purchase 250 180
Sales Uplift 575,000 18,000
To work out the total amount that my target group (population) would have spend if we did not run the campaign, I would do:
10,000 x 20% x 180 = $360,000
To work out the uplift I would do:
(575,000 + 18,000) - 360,000 = $233,000
So my questions are:
How does is the above calculation effected by statistical significance?
For example, if I find that there is no statistical significance between 24% and 20% (hypothetically). Does that mean that doing this calculation is no longer relevant since the campaign did not have a significant impact on the proportions?
Similarly, what if there is no statistical difference between the Average purchase amount?
What about if there is a significant difference in proportion but not significant difference in average purchase?
Let's say that the data is highly skewed and does not adhere to assumptions for two-sample t-test (purchase amount). I would then use a non-parametric test to look for statistical significance. Let say I find that there is a statistical significance.
Is there a way to use median instead of mean for the above calculation?
Or is it still ok to use the mean purchase amount (even though it's skewed) because we are trying to get to the total spend of the entire population without the campaign?
In other words, it doesn't matter that the data is highly skewed, and that the mean is very different from the median. We want to know the total dollar uplift, not the middle value of spend. Therefore, we can use the average figure.