6
$\begingroup$

I have 2 groups of data to compare using a t-test, both of sample size n=5. Checking to see if the normality assumption of the test holds is difficult since the sample sizes are too small.

I have read that a t-test is robust to non-normality in this case since the sample sizes are equal. However if the data were highly non-normal it might no longer be robust. An alternative would be to perform an Mann-Whitney U test but this test has less power than the t-test (though I'm not sure how less). I actually encounter situations similar to these all of the time and realise there may not be a definitive answer but I am just curious how others in the community would approach this sort of problem. I typically just go with the t-test and tell others to do the same since they are more familiar with the test than the MW test.

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ "but this test has less power than the t-test" -- sure, at the normal, where it has about 5 percent lower ARE - that is, when all the assumptions of the t-test are exactly met, you need 5% more observations (in large samples) for the MW to have the same power for small effect size as the $t$. However, as the distributions become even mildly heavy-tailed, the MW becomes more efficient than the $t$... and when tails get heavy, it can be arbitrarily more efficient than the $t$. In the presence of discreteness and skewness the differences are less clear. $\endgroup$
    – Glen_b
    Jun 27, 2013 at 6:11
  • $\begingroup$ Heteroskedasticity complicates the comparisons between them as well; it depends more on your specific alternatives. If the distribution is lighter-tailed than the normal, the $t$ advantage is bigger, but rarely very big. One problem with MW in really small samples may be the inability to reach really small p-values. With two samples of 5 you should just be okay (your smallest achievable p-value is about 0.8%). $\endgroup$
    – Glen_b
    Jun 27, 2013 at 6:17
  • $\begingroup$ Thanks Glen, it seems to me that with all of the conditions you mentioned, the situation involving the power of the two tests is much more complex than I thought. Thanks for that help. Maybe will just leave it at this- there isn't a clear winner in terms of which test to use in this case and both tests are reasonable. $\endgroup$
    – Jimj
    Jun 27, 2013 at 7:30
  • 7
    $\begingroup$ I think my discussion is less equivocal than you think. (1) the t-test can be arbitrarily bad compared to the MW, the converse does not hold; (2) the MW is very nearly as good as the t, when its assumptions hold exactly; (3) the case where the $t$ has a (moderate) advantage - lighter tails - is rarer than the case where it can have a large disadvantage (heavy tails); (4) with small samples you can't check how heavy tailed it is, so just you don't know quite how badly you're screwed; (5) finally, there are a few side cases where it's not especially clear which is better. $\endgroup$
    – Glen_b
    Jun 27, 2013 at 7:38
  • 2
    $\begingroup$ Although you got no answers, you got several highly pertinent and closely-argumed comments from @Glen_b. Posting a bounty implies that you think more can be said, but what specifically do you seek? $\endgroup$
    – Nick Cox
    Jun 30, 2013 at 12:40

2 Answers 2

Reset to default
4
+25
$\begingroup$

How I would approach this: I would not go for a t-test if the requirements for applying it are not (or not known to be) satisfied. This seems obvious to me and good practice for any hypothesis testing. Mere familiarity with one or other test above other one(s) is not a justification. Can you give details where you found that the difference in centrality two small samples from unknown and unverifiable population distributions would be robust against non-normality? I find this hard to accept. Nonparametric tests like M-W compare the difference of population medians (as opposed to mean values), because the median is more robust. This is especially true in your case, where you cannot test for the population distribution. Do you have any prior information or evidence that the two samples were drawn from the same population? Perhaps a description of the experiment might help to judge. You may be pushing the boundaries about what hypothesis testing can do for you here. With small samples (and no repeat experiments available, I presume?) and without any information on the population, I would not want to involve & quantify statistical concepts like power, significance, CI, etc. All you can do really is list some descriptives for the two samples (quote the two medians or their difference, relative to their sum; the ranges & maximum range of their difference, etc.). However, if you could repeat by drawing many samples of size n=5, that would change the picture at lot. Is that an option for you?

Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ Nice comment but standard np tests do not compare medians. They relate to the Hodges-Lehmann estimator and pseudomedians (median of all possible pairwise differences). $\endgroup$
    – Frank Harrell
    Jul 1, 2013 at 14:01
  • $\begingroup$ I have read that some people choose the t-test in situations such as these because they don't have evidence that the assumptions aren't satisfied. $\endgroup$
    – Jimj
    Jul 2, 2013 at 8:23
  • $\begingroup$ @Jimj: OK, but that is then a matter of how much risk as type II error you can accept -- in the style of "innocent until proven guilty" or vice versa. Food for Bayesians, in any case. $\endgroup$
    – Lucozade
    Jul 2, 2013 at 15:24
4
$\begingroup$

You should definitely consider a permutation test, as you can use the mean as test statistic and have a lot less assumptions. You will find a lot of information if you google permutation test or search it here. An implementation in R (taken from this great answer to one of my questions):

a <- rnorm(5)
b <- rnorm(5, 0.5)
DV <- c(a, b)

ids <- seq(along = DV)          # indices to permute
idx <- combn(ids, length(a))    # all possibilities for different groups

# function to calculate difference in group means given index vector for group A
getDiffM <- function(x) { mean(DV[x]) - mean(DV[!(ids %in% x)]) }    
resDM    <- apply(idx, 2, getDiffM)      # difference in means for all permutations
diffM    <- mean(a) - mean(b)           # empirical differencen in group means

# p-value: proportion of group means at least as extreme as observed one
(pVal <- sum(resDM >= diffM) / length(resDM))
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.