I have three groups. Both have small sample sizes (<10). And we can't check the assumption of normality. I want to compare any two of them.

What to do in this case( unequal small sizes, non-normality)

  • $\begingroup$ How about a permutation test? You may not have much here with such small samples, but on the plus side you can do full enumeration. $\endgroup$ – soakley Dec 7 '13 at 23:27
  • $\begingroup$ I don't understand. How to carry out a permutation test ? What's its assumptions? $\endgroup$ – OMG Dec 7 '13 at 23:32
  • $\begingroup$ So much I want to say about this, but first some questions -- when you say 'I want to compare them' - what is it you're trying to compare? Do you mean to compare locations? Is there a specific alternative? Are you talking about 3 pairwise comparisons or are you after something notionally more like a one-way ANOVA? (The suitability of permutation tests will be affected by how small the sample sizes are -- if they're small enough, you won't be able to get below some specific significance levels; on the other hand, outside specific parametric assumptions you may not have much choice about that) $\endgroup$ – Glen_b -Reinstate Monica Dec 7 '13 at 23:55
  • $\begingroup$ A good description of a permutation test will not fit in a comment. But check Wikipedia under "Resampling" for a decent start. $\endgroup$ – soakley Dec 8 '13 at 0:13
  • 1
    $\begingroup$ So does that mean you want to change your question to 'how do I do multiple comparisons after an ANOVA?' or do you actually want to know something like 'how do I do something like an ANOVA and multiple comparisons but without assuming normality?' ...? (Incidentally, what are your sample sizes?) $\endgroup$ – Glen_b -Reinstate Monica Dec 8 '13 at 1:13

(This is too long for a comment; if the question is sufficiently clarified - see the comments - it may be edited into a better answer.)

On permutation tests:

The basic idea of a permutation test is that if the null hypothesis is true, the group/category labels are arbitrary and could be reassigned without changing the distribution of the groups. So they are reassigned, in every fashion and a test statistic sensitive to the alternative is calculated; if the null is true, the sample should be a randomly chosen member of that permutation distribution. If the alternative is true, the sample should tend toward being 'extreme' in that distribution (extreme in a sense defined by the ordering resulting from what is most consistent with the alternative).

There's some discussion of permutation tests here, discussion of why they might be used here, discussion of the relationship between permutation tests and randomization tests here, and a little on the choice between permutation tests and t-tests here.

There are examples of different permutation tests done here and here and here and here (see near the end).


No matter what you use your power to detect differences will be low. Another possible approach would be using non-parametric tests. If you have three groups you could use Kruskal-Wallis (followed by a multiple comparison procedure if you want to compare them pairwise).

In R you can use kruskal.test for the Kruskall Wallis test and kruskalmc in the pgirmess package for multiple comparisons

  • $\begingroup$ No matter what you use your power to detect differences will be low How to understand this? $\endgroup$ – OMG Dec 11 '13 at 20:01
  • $\begingroup$ OMG: With sample size so small, the power will be low. $\endgroup$ – Glen_b -Reinstate Monica Dec 11 '13 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.