So let's say I did an experiment comparing mutated samples vs non mutated where I have an unequal number of samples (100s (non mutated) vs 5-10 (mutated)). I think the groups might have different variances so I could do a welch t-test to address that issue. But a colleague thinks doing permutation testing and drawing sample size equal to the smallest group would be better. While I think in some cases permutations are great but in this case I'm not so sure. Also looking at his R code I'm not sure it's correct.. shouldn't you be sampling from each group independently rather than pooling them

# groupA is mutated samples and groupB is non mutated samples
groupA <- which(samples[,i] == "mutated")
groupB <- which(samples[,i] == "non_mutated")
sample.size <- length(groupA)
groupAB <- c(groupA,groupB)

for(j in seq(1:length(colnames(results))))
    actual.med.diff <- median(na.omit(results[groupA,j])) - median(na.omit(results[groupB,j]))

    for(k in 1:1000)
        index <- sample(1:length(groupAB), size=sample.size, replace=FALSE)
        perm.groupA <- groupAB[index]
        perm.groupB <- groupAB[-index]
        perm.med.diffs[k] <- median(na.omit(results[perm.groupA,j])) median(na.omit(results[perm.groupB,j]))
    median.perm.pval <- length(which(perm.med.diffs <= actual.med.diff)) / 1000

Permutation testing is a good way to go with a group with few observations; however, you are not quite doing it right. You don't choose the sample size of the permutation draw, rather you permute the labels for your drawn sample. for instance if you observe:

outcome <- rnorm(100)
group <- sample(c("a","b"), 100, prob=c(.1,.9),replace=TRUE)
medianDifference <- abs(diff(tapply(outcome,group,median)))

One permutation draw would look like

groupPerm <- sample(group, length(group))
permMedianDifference <- abs(diff(tapply(outcome,groupPerm,median)))

To get your p-value, you find the percentage of times permMedianDifference > medianDifference

  • $\begingroup$ Thanks Ian, yes I realized after I posted it what looked wrong and it was the permuting the labels. What about the assumption of the difference in distributions $\endgroup$ – steve Mar 31 '16 at 20:07

A problem with doing a permutation test of equality of means occurs if you think that the possible difference in variance of the two groups will remain even if the null of equal means is true. In that case the label assignments are not arbitrary when the null is true (the observations are not exchangeable) – since you'd appear to have no basis to argue that you can swap labels under the null, the reasoning behind the permutation test wouldn't seem to hold.

[Perhaps there's some other argument for being able to permute the labels under the null, but I don't know what it is.]

However, imagine instead that you have a case where you posit that mutation either has no effect on the distribution (the null case), or some effect on the distribution (impacting both mean and variance at the same time, say).

In that case, under the null you can permute (since the labels really are arbitrary at the null). But under those assumptions, you could also carry out an equal-variance t-test (as long as the assumption of normality under the null was reasonable enough that you're happy with the resulting test properties); the fact that equality-of-variance is violated under the alternative doesn't impact the correctness of the significance level nor of p-values (though it would impact power, so if you're relying on power calculations for sample size, you would definitely worry about it then – simulation would be one way of getting a sense of what the power could be like under various relative variances at any given difference in means).

  • 1
    $\begingroup$ You make a good point that is often overlooked in talking about "non-parametric" tests. The H0 for permutation based tests is always H0: f(x) = g(x), where f and g are the distributions of the two groups, so differences in variance can reject the null more than expected. That said, if you permute the labels and calculate the p-value using the welsh t-statistic, then as the sample size grows larger the statistic becomes ancillary to changes in variance. This way you get the best of both worlds, a valid test of H0: f(x)=g(x) for small samples and a valid test for H0:mu_1=mu_2 for large. $\endgroup$ – Ian Fellows Apr 1 '16 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.