Is this an Appropriate Application of a Permutation Test?

Question

Suppose I have count data between three groups, each with a different number of observations: $n_1 = 11, n_2 = 6$ and $n_3 = 5$. My data is listed below:

\begin{array} {|r|r|}\hline group 1 & group 2 & group 3 \\ \hline 9 & 8 & 6 \\ \hline 9 & 8 & 36 \\ \hline 8 & 7 & 10 \\ \hline 11 & 0 & 10 \\ \hline 18 & 0 & 6 \\ \hline 4 & 4 & \\ \hline 0 & & \\ \hline 0 & & \\ \hline 8 & & \\ \hline 12 & & \\ \hline 12 & & \\ \hline \end{array}

I'm interested in determining if there is a meaningful difference between the means of these three groups. The observations are all independent, both within and between groups. Each observation can take on a value of $0-36$. I plotted the density functions for each group of observations, and (as is to be expected for discrete data with relatively few observations per group), they do not appear to be normally distributed and neither do their residuals. So I cannot simply apply one-way ANOVA.

I had a tough time trying to find a discrete parametric model under which I could properly fit the data. So, I figured that perhaps a permutation test using the F-statistic may provide me with a reasonable approach to determining if the means of any one of these 3 groups are significantly different. upon plotting the distribution of my test statistic after 5000 itterations, it closely resembled the theoretical F distribution for $df_1=2,df_2=19$.

To my understanding, the p-value I obtained by running this permutation test represents the probability of obtaining an observed test statistic as large as my initial test statistic (ie. the F-stat calculated from my original data set) assuming the null hypothesis to be true. Given that my p-value was somewhat large, $0.131$, I can conclude that there isn't significant evidence against the null hypothesis and that the means between these 3 groups do not differ significantly.

Is my interpretation of this permutation test correct?

Answer 1

Yes, a permutation test makes sense here.

Strictly speaking, the null hypothesis is that the distributions are the same, not just that they have the same means. (If they had same means but difference variances, the test would have the wrong Type I error rate.)

Here's example code

> df<-data.frame(group=factor(rep(c(1,2,3),c(11,6,5))),
+               count=c(9,9,8,11,18,4,0,0,
+               8,12,12,8,8,7,0,0,4,
+               6,36,10,10,6))
> obsF<-summary(aov(count~group,data=df))[[1]]$"F value"[1]
> 
> 
> Fs<-replicate(50000,{
+ i<-sample(nrow(df))
+ summary(aov(count~group[i],data=df))[[1]]$"F value"[1]
+ })
> 
> qqplot(qf((1:50000)/50001,2, 19),Fs, xlab="Expected",ylab="Simulation")
> abline(0,1)
> mean(Fs>=obsF)
[1] 0.12894

As you see, I get 0.129, which agrees with what you got up to Monte Carlo uncertainty.

A quantile-quantile plot against the $F^2_{19}$ distribution shows good agreement for small values of the statistic, but less good at large values -- it's harder to get very high $F$ statistics than it would be with iid Normal data.

The agreement isn't surprising: in two-sample testing the permutation distribution of the (two-tailed) $t$ statistic is close to a $t$ distribution whether or not the assumptions are satisfied, and this is the same sort of thing.

I might have tried ANOVA on the square roots of the counts instead, to try to stabilise the variance-mean relationship at bit. In these data this gives essentially the same $p$-value, though.