0
$\begingroup$

I have a large dataset (n = 170,000), distributed roughly equally among 5 groups (each is 34,000 ± ~1,000). I want to compare whether the means in any of the first 4 groups are different from the control. Before the data came in, I figured the best approach would be to use Dunnett's test; however, now I'm unsure how best to handle this.

After doing some research, it seems there are at least 2 ways to go about it:

  1. Normalize the values (which still yields a skewed distribution)
  2. Use a non-parametric counterpart

I'd like to get an idea of the pros/cons of each, so I can decide on the best approach.

$\endgroup$
7
  • 2
    $\begingroup$ It's quite likely a standard ANOVA is the best procedure, but that depends on which "distribution" the title refers to--is it the distribution of the responses of of their residuals?--and how exactly it departs from Normality. Could you provide this important information by editing your question? $\endgroup$
    – whuber
    Commented Apr 30, 2018 at 20:59
  • $\begingroup$ Thanks for the comment, whuber. My understanding was that, since Dunnett's compares the means of the control to the treatments, it's the distribution of the dependent variable that needs to be normally distributed. Did I misunderstand that? $\endgroup$
    – Khashir
    Commented Apr 30, 2018 at 21:14
  • $\begingroup$ What needs to be close to Normally distributed is the sampling distribution of the difference of means. With very large groups it would take extreme skewnesses to violate that assumption to the degree it would affect any but tiny p-values. $\endgroup$
    – whuber
    Commented Apr 30, 2018 at 22:37
  • $\begingroup$ What do you mean by "normalize"? $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 0:36
  • 2
    $\begingroup$ How would that help? You'd remove any differences in mean (which is what you're interested in), without removing the skewness that you're trying to deal with. $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 6:24

1 Answer 1

3
$\begingroup$

You have several considerations.

  1. Significance level. Unless you have extremely skewed distributions, correctness of significance level is probably not at issue if the group-variances don't differ very much.

    If significance level was still a concern for you, you could avoid that problem by simply performing a permutation test, perhaps with the usual statistic or some suitable simplification of it.

    If variances differ, that will affect significance level, but a Welch-Satterthwaite approach may be adequate to deal with that.

  2. Power. Power may indeed be an issue. You might want to consider a more suitable distributional model (e.g. possibly an exponential-family model -- a GLM -- which will still allow you to compare means). What sort of quantity are you measuring? Are these times? Incomes? Counts? Angles?

  3. A rank-based nonparametric test will not be a test for equality of means without additional assumptions (assumptions that would keep the means equal under the null and unequal under the alternative). If such assumptions don't hold, you can have equal means and yet be highly likely to reject, or unequal means and be highly likely to fail to reject, even in very large samples.

$\endgroup$
8
  • $\begingroup$ Heya, thanks for submitting an answer. We are measuring average counts (e.g., pizza slices per person, for example). $\endgroup$
    – Khashir
    Commented May 1, 2018 at 1:07
  • 2
    $\begingroup$ if you know the denominators in your average counts, why not use a model for counts? A Poisson or negative binomial regression model, perhaps (there are a number of other alternatives). These would give you a model for the mean, as you require, and the ability to carry out a test that would correspond to the kind of comparison of means you're after (fit a full and reduced model, use an asymptotic chi-squared test). A suitable choice may deal with the skewness and the heteroskedasticity you definitely expect with count data all at the same time (those aspects are in the distributional model). $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 6:22
  • $\begingroup$ Hmm, this might be beyond me at the moment. Would you have a resource for me to dive in, or a useful query to throw at google? I want to have a good grasp of what I'd be doing in that case. $\endgroup$
    – Khashir
    Commented May 1, 2018 at 14:21
  • 1
    $\begingroup$ I'd start with googling generalized linear model which extend regression models to a much wider class of practical models; here's Wikipedia on GLMs. Also here's lecture slides for an introductory lecture on them (don't worry about the stuff on estimation methods), but it focuses on using R to fit them, which may not suit you but it will give some of the gist. It might be a little more mathematical than you want but you may find parts of it useful. ...ctd $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 22:37
  • 1
    $\begingroup$ ctd... John Fox's book (applied regression analysis and generalized linear models) isn't bad, or you may prefer a specific book on GLMs. I'd also search for poisson regression and - eventually - for negative binomial regression. There are many questions here on site about GLMs and on Poisson and negative binomial regression models including a fair number of introductory questions and some have good answers. Strictly speaking negative binomial models are not GLMs unless you specify a parameter but it's often treated as one; learning GLMs/Poisson regression will be valuable as a 1st step $\endgroup$
    – Glen_b
    Commented May 1, 2018 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.