4
$\begingroup$

Wondering if anyone has an opinion on whether bootstrapping the difference in means is the right method given I have a situation with extreme data points. I've decided to use this as I don't think a t test is appropriate

I have about 30k observations per group (3 groups)

My situation is about spend, and I have extreme outliers: the outliers aren't quite like an "income" distribution. That is, most users (95%+) will spend zero, a subset of users will spend 5 - 10 dollars. some will spend about 20 or 50 dollars and then a select few will spend 500+, with a couple of users spending 5000 or 10000+

I am trying to test which group brought in the most revenue per user.

Can anyone offer any advice on which statistical test is best suited?

Improve this question
$\endgroup$
4
  • $\begingroup$ Your client should just concentrate on pursuing these wealthy guys to buy/contribute, and this is a PR question, not a statistics question. $\endgroup$
    – StasK
    Commented Oct 9, 2013 at 13:53
  • 1
    $\begingroup$ Would some GLM with a sufficiently right-skew distribution - perhaps gamma or inverse-Gaussian - solve this problem? $\endgroup$
    – Glen_b
    Commented Nov 19, 2013 at 22:18
  • $\begingroup$ Maybe first estimate the probability of being a spender for each group, and then compute group means conditional on being a spender. Does anyone know of a reason why that wouldn't work? Or is this just a tobit model? $\endgroup$
    – shadowtalker
    Commented Dec 20, 2013 at 3:01
  • $\begingroup$ Sorry if this is a dumb question, but with so many observations, why not just use an ANOVA (or a t-test if you want to compare 2 groups)? Doesn't the central limit theorem hold here? What am I missing? $\endgroup$
    – digbyterrell
    Commented Dec 20, 2013 at 15:16

3 Answers 3

Reset to default
3
$\begingroup$

Your data seem to have very heavy tails: Few very high expenses. In this case, bootstrapping the means is not a good idea. In fact, t-test (and friends, e.g. ANOVA) isn't either. Maybe you could take Janssen and Pauls, Ann. Stat. (2003) as a starting point to read more about all this.

I would suggest to rank the expenses and make any inferences depend on the ranked data, e.g. Kruskal-Wallis-Test in your case. So the few very high expenses will have less impact on your result. This may result in a loss of power, if your data would have supported t-tests, but as you have really huge samples, your test may be powerful enough. OT: In fact, your test may even be overpowered, that means you can detect differences that are too small to be of any practical impact, e.g. if the difference in the mean expenses is less than 1 Cent. But you did not ask about tests of relevance.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ To avoid problems of practical significance and statistical significance, you could use an effect size metric. I would recommend something like the common language effect size (CLES). This is rank based and can be interpreted as the probability of a random individual in group A being larger than a random individual in group B. $\endgroup$
    – dmartin
    Commented Dec 20, 2013 at 16:11
2
$\begingroup$

It is easy to overcomplicate a question such as this.

"Which group brought in the most revenue per user?" is a factual question and you have exact revenue numbers.

There are no hypotheses to be tested.

You simply divide the group revenue by the group size and compare.

It is when you add some noise into the system (e.g. sales may be misreported by up to 20% on each sale, or might be embezzled randomly) or ask predictive questions that imply some sort of noisy error process (e.g. "if we repeated something like this again, what would happen?"), that you need to deal with statistical issues.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The question is also whether the difference is statistically significant. $\endgroup$
    – Tomek Tarczynski
    Commented Jan 5, 2014 at 12:16
1
$\begingroup$

Set aside the theoretical issues related to extreme observations, I would not advise ttest anyway, as you seem to have 3 groups of observations. Hence, you would be probably better with ANOVA and subsequent multiple comparisons.

Moreover, as far as your zero-spenders are concerned, I would try to understand how many of them are systematic zeros (that is, people who cannot afford to spend money because they cannot rely upon a disposable income) vs sample zeros (that is, people who are not interested in buying the goods considered in your research).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why would the OP set aside extreme observations? They're likely to have a substantial effect on the mean, which is what they want to compare! $\endgroup$
    – Glen_b
    Commented Nov 19, 2013 at 22:17
  • 1
    $\begingroup$ I was unclear in my previous reply, as Glen_b highlights. Hence, I have edited the first sentence of my answer as follows: "Set aside the theoretical issues related to extreme observations..." $\endgroup$
    – Carlo Lazzaro
    Commented Jan 5, 2014 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.