I have a preliminary study with very small sample size (n=26), and I want to test for differences between males and females and similar things, so I have to divide the sample and make comparisons of 13 vs 13 subjects.

Is there a test I can use to give an idea of what the differences might be? Is it possible to use the two-independent-sample t-test even if the sample is this small and the variable is not normally distributed? What can I do otherwise?

  • $\begingroup$ Have you tried transforming the data already? If you can transform it to normally distributed, that would be good! $\endgroup$
    – Kyle.
    Nov 4, 2012 at 17:20
  • 1
    $\begingroup$ You can try a permutation test. $\endgroup$
    – Peter Flom
    Nov 4, 2012 at 17:27
  • $\begingroup$ You might find this article in American Psychologist useful. $\endgroup$
    – RioRaider
    Nov 4, 2012 at 17:39
  • $\begingroup$ @Kyle but then you're no longer testing for a location difference, nor are you testing means, on the original, untransformed scale. $\endgroup$
    – Glen_b
    Nov 4, 2012 at 21:41
  • $\begingroup$ @Kyle To clarify; that may not be a problem, depending on what tracy wants, but I thought it was important to be clear about what was actually being tested. If detecting general kinds of increase/decrease is okay, one may as well choose a scale that's invariant to monotonic transformation (which would be a test based on ranks). $\endgroup$
    – Glen_b
    Nov 4, 2012 at 21:56

1 Answer 1


Since your dataset is small, you are in the fortunate situation where you can calculate all possible partitions into two groups.

In your case, you want two groups of size 13, so you have "26 choose 13" combinations, which is about 10.4 million combinations. Now, for every given statistic you want to test (it could be mean but doesn't have to be), you can go over all combinations and count in how many of them the statistic was equal or higher than what you observed in your partition. There is of course a theoretical limit on the p-value, which is 1 over the number of combinations. If you have some minimal programming skills this should be easy to implement (10.4 million combinations would run very fast on any modern computer) and would give you the most accurate results without assuming much.

Another option, but less robust, is to use a permutation test which means the same as above but instead of explicitly calculating all combinations, sampling from them randomly. This will be less accurate and will limit your p-value to the number of permutations you try. Also, beyond some number of permutations these results will become less accurate because of the limited number of possible combinations.

  • $\begingroup$ @tracy A permutation test of the means of the data-transformed-to-ranks is just a two-sample Mann-Whitney-Wilcoxon test; if the data are reasonably close to normal the permutation test will have better power to detect differences, otherwise - e.g. if the tails are heavier - it may not. (Personally, I'd be inclined to do the full permutation test on the means as Bitwise suggests, because it's explicitly testing for a difference in means.) $\endgroup$
    – Glen_b
    Nov 4, 2012 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.