Permutation testing

Question

So let's say I did an experiment comparing mutated samples vs non mutated where I have an unequal number of samples (100s (non mutated) vs 5-10 (mutated)). I think the groups might have different variances so I could do a welch t-test to address that issue. But a colleague thinks doing permutation testing and drawing sample size equal to the smallest group would be better. While I think in some cases permutations are great but in this case I'm not so sure. Also looking at his R code I'm not sure it's correct.. shouldn't you be sampling from each group independently rather than pooling them

# groupA is mutated samples and groupB is non mutated samples
groupA <- which(samples[,i] == "mutated")
groupB <- which(samples[,i] == "non_mutated")
sample.size <- length(groupA)
groupAB <- c(groupA,groupB)

for(j in seq(1:length(colnames(results))))
{
    actual.med.diff <- median(na.omit(results[groupA,j])) - median(na.omit(results[groupB,j]))

    for(k in 1:1000)
    {
        index <- sample(1:length(groupAB), size=sample.size, replace=FALSE)
        perm.groupA <- groupAB[index]
        perm.groupB <- groupAB[-index]
        perm.med.diffs[k] <- median(na.omit(results[perm.groupA,j])) median(na.omit(results[perm.groupB,j]))
    }
    median.perm.pval <- length(which(perm.med.diffs <= actual.med.diff)) / 1000
}

Answer 1

Permutation testing is a good way to go with a group with few observations; however, you are not quite doing it right. You don't choose the sample size of the permutation draw, rather you permute the labels for your drawn sample. for instance if you observe:

outcome <- rnorm(100)
group <- sample(c("a","b"), 100, prob=c(.1,.9),replace=TRUE)
medianDifference <- abs(diff(tapply(outcome,group,median)))

One permutation draw would look like

groupPerm <- sample(group, length(group))
permMedianDifference <- abs(diff(tapply(outcome,groupPerm,median)))

To get your p-value, you find the percentage of times permMedianDifference > medianDifference

Answer 2

A problem with doing a permutation test of equality of means occurs if you think that the possible difference in variance of the two groups will remain even if the null of equal means is true. In that case the label assignments are not arbitrary when the null is true (the observations are not exchangeable) – since you'd appear to have no basis to argue that you can swap labels under the null, the reasoning behind the permutation test wouldn't seem to hold.

[Perhaps there's some other argument for being able to permute the labels under the null, but I don't know what it is.]

However, imagine instead that you have a case where you posit that mutation either has no effect on the distribution (the null case), or some effect on the distribution (impacting both mean and variance at the same time, say).

In that case, under the null you can permute (since the labels really are arbitrary at the null). But under those assumptions, you could also carry out an equal-variance t-test (as long as the assumption of normality under the null was reasonable enough that you're happy with the resulting test properties); the fact that equality-of-variance is violated under the alternative doesn't impact the correctness of the significance level nor of p-values (though it would impact power, so if you're relying on power calculations for sample size, you would definitely worry about it then – simulation would be one way of getting a sense of what the power could be like under various relative variances at any given difference in means).