3
$\begingroup$

I have a paper under review where I conducted a Kruskal Wallis test in SPSS to compare 4 groups. Each group contains payments made by companies over one year, and group 1 received substantially more payments than the other groups.

Group 1 - 18,190
Group 2 - 1,370
Group 3 - 990
Group 4 - 311

A reviewer suggested that for group 1, I should use a random 10% of the sample, however, I cannot find references to support doing this. I think removing 90% of the Group 1 sample would ignore most of the data.

Is anyone able to advise whether it is acceptable to run a KW on 4 very different groups?

$\endgroup$
2
  • 1
    $\begingroup$ Part of an answer: I wouldn't re-sample the data. ... In general, with sample sizes that large, you are likely to find a significant result even for small differences. Be sure to look at effect size statistics or summary statistics or plots to see if the differences you find are practically important. $\endgroup$ Commented Dec 22, 2022 at 13:32
  • 1
    $\begingroup$ There is an over-arching question of whether some other test or even descriptive comparison comes closer to your goals. For example, it would usually make as much or more sense to analyse payments on a logarithmic scale (and zero values are not an absolute barrier to this: just use a generalized linear model with logarithmic link). Further, I rarely find K-W tests illuminating. They were historically designed to do a good honest job with very small datasets, not for this kind and size of problem. $\endgroup$
    – Nick Cox
    Commented Dec 23, 2022 at 12:25

1 Answer 1

5
$\begingroup$

This is related to a disturbing but very common misunderstanding.

Tests for one-way type comparisons (one-sample location tests) are designed for different sample sizes. There's no harm in the sample sizes being different, as long as the assumptions are reasonable (beware, also, choosing your tests based on what you find in the data; this can sometimes cause the sort of problem you're trying to avoid).

If you're designing an experiment or some such, it's best to allocate the total sample size equally to each group but if you have sample sizes that are unequal, that preference is not a consideration.

  • you lose power by reducing sample size in the large groups, for no obvious gain anywhere else

  • if you're throwing away information at random, you also get the very strange property that two people using exactly the same test on exactly the same original data may come to different conclusions. This is not my idea of a desirable property.

  • if robustness to unequal spreads is an issue (which can matter if it occurs under H0), you'd be better modifying your choice of test than throwing away data. You cannot typically judge whether this is an issue from the data (since the null is likely false and spread that changes as the location changes could easily produce differing spreads under H1). Beware taking drastic action to compensate for this unnecessarily; this may be considerably less likely to be a problem than many people seem to think.

advise whether it is acceptable

Acceptable to whom? There's not necessarily any statistical issue, but that doesn't mean a colleague (e.g. a reviewer) won't cause you problems; they may even point to papers in your application area (whatever it might be) that make the same sort of claim.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.