0
$\begingroup$

I'm trying to analyse the impact of a marketing campaign and I'm looking to identify the most appropriate statistical test. I have one group of customers who received the campaign (approx 100,000 people) and a random control (approx 50,000 customers) who received nothing. I want to measure whether there is any difference in the average spend from both groups in the period following the campaign launch.

The vast majority of customers in both groups will spend nothing after the campaign. Only approximately 2% of the customers in the groups will spend anything. So the average spend variable in both groups is not normally distributed - it will have a huge tail at 0, corresponding to the large number of people who will spend nothing.

As the population sizes are very large (100,000 and 50,000) can I just use a 2 sample z test as normal due to the central limit theorem? Or will the fact that the variable being tested is highly skewed mean that a 2 sample z test is invalid?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ That's not what the CLT says. Z-test requires distributions to be normal. $\endgroup$ – user2974951 Oct 10 '18 at 13:45
  • 1
    $\begingroup$ @user2974951 - with 50,000 observations, the distribution of the sample mean may very well be so close to Gaussian (thanks to the CLT) that a z-test will work fine, even with 98% of the observations being 0. $\endgroup$ – jbowman Oct 10 '18 at 16:29
2
$\begingroup$

As noted in a comment, z-test is not appropriate in this case, since it requires a normal distribution and your data is obviously not normally distributed.

EDIT: When can you use z-test? It is true that many test statistics may approach normality with large number of samples. Now the question is whether your mean amount spent will approach a normal distribution given the large number of your samples. The way you have described (with majority, 98% of individuals spending nothing), it might be contested. I am at heart an experimentalist, so here is what I'd do: draw samples of 10,000 from your data and see whether the means are normally distributed on a q-q plot. If your data is good enough at 10K samples, it will be even better at 50K.

One possibility is, of course, a non-parametric alternative such as a U-test. With these numbers, you could also safely use a randomization approach.

However, it might be that whatever the obtained p-value, your results will not be very meaningful – precisely because of the large sample size, which will allow to detect even very small effects. If the amount spent is, say, in \$10-20 range, would an average difference of \$0.05 make any difference?

In other words, your statistical test may be overpowered.

Rather than focusing on the test, you could focus on the effect size – or actual difference (and estimate its confidence intervals).

Another problem with your approach is that very often the amount spent is not a continuous variable; for example, we are talking about a price list which contains a finite number of prices (e.g., \$9.99, \$19.99 and \$29.99, and no-one ever spends \$12.75). If that is the case, maybe comparing the frequencies of individuals who choose a particular option (or none at all) using a χ² test or something similar might be a better choice.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks - there appears to be conflicting advice on this online. I've read other advice saying when sample sizes are high a z-test or t-test approach is fine. For the U test, how would I handle all the 0 values when the records are ranked? As they make up say 98% of the records would they all get give n the same rank? $\endgroup$ – Alistair A Oct 10 '18 at 15:24
  • 1
    $\begingroup$ Well, yeah, that's the question. Normally, I would say that you are right, but I wonder whether with such a large fraction of 0's you will approach normality at your sample size. I have included an edit to my answer. $\endgroup$ – January Oct 11 '18 at 8:30
  • $\begingroup$ Thanks for the advice. The bit I'm struggling with is that these groups aren't really "samples" from populations. They are populations - the test group is everyone who got the campaign. The control group is "everyone else" who could have received the campaign but didn't due to a random selection (pre-campaign the groups are identical). From the answers so far, after checking sample means for normality as suggested, I'm unsure if I should do a) a z test on the 100k vs the 50k or b) take large samples (e.g. 40k) from both groups and do a z test / t test on those. Which is the better approach? $\endgroup$ – Alistair A Oct 11 '18 at 8:43
  • 1
    $\begingroup$ They are samples, because you want to infer about future clients (the full population) and not merely about the people you have in your experiment. If I take 3 mice and give them a novel drug, and compare it to the 3 mice that did not get the drug, I could in theory treat these 6 mice as full populations (after all, no other mice in the world received my drug), but the whole purpose of my statistics is to make inference about what happens when I give that drug to any other mice. $\endgroup$ – January Oct 11 '18 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.