2
$\begingroup$

I have to compare three groups (each group is a customer to a subscription box company).

Group A received treatment A. Group B received treatment B. Group C received no treatment.

We count the total number of boxes sold in each group (i.e. we are modelling count data)

I want to compare the mean number of boxes in group A and B vs the control (to see if the uplift is greater)

For this data ~2% of Group A got a box, ~2.1% of Group B got a box, and ~1.1% of Group C got a box.

What tests could we run to compare the means or the difference in uplift vs control for Group A and Group B?

I don't think a t-test or a Mann-Whitney test is appropriate here (though I am willing to be proven wrong!)

Improve this question
$\endgroup$
10
  • $\begingroup$ Run a zero inflated negative binomial model using group c as the reference. $\endgroup$
    – Demetri Pananos
    Commented Oct 28, 2019 at 22:34
  • $\begingroup$ Why isn't the outcome binary here if the data is at user level? Can users purchase more than one subscription box? $\endgroup$
    – dimitriy
    Commented Oct 28, 2019 at 23:39
  • $\begingroup$ Why do you think this is zero inflated data as opposed to just data with zeroes in it? $\endgroup$
    – jbowman
    Commented Oct 29, 2019 at 1:58
  • $\begingroup$ @DimitriyV.Masterov users can buy a box a week, and we look at how many boxes they bought in a 20-week window. The outcome would be binary if it was just conversions after 20 weeks. $\endgroup$
    – Tom Kealy
    Commented Oct 29, 2019 at 8:47
  • $\begingroup$ @jbowman I'm not sure of the distinction, I'm afraid. $\endgroup$
    – Tom Kealy
    Commented Oct 29, 2019 at 8:48

1 Answer 1

Reset to default
3
$\begingroup$

We are to gather "uplift" is basically number of boxes. If you want to compare the mean number of boxes you should use an ANOVA. If the sample size is reasonably large, the sampling distribution of the mean is approximately normally distributed.

Models for count data can be used as well, given that the outcome is count. Poisson regression is the most common. Having many 0s does not mean the data are not Poisson, the rate could just be low. Quasipoisson and negative binomial models both just scale the variance so that the mean is merely proportional to the variance, in all cases the effect is interpreted as a relative rate of number of boxes.

A rank based test will tell you nothing about the mean. Rank tests in general are not a panacea for violations of modeling assumptions (which is separate from "having lots of zeroes"). Inferring differences in mean does not require that an exact parametric model is specified, rather using robust or asymptotic statistics will give you valid inference about mean differences.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.