Suppose I have count data between three groups, each with a different number of observations: $n_1 = 11, n_2 = 6$ and $n_3 = 5$. My data is listed below:

\begin{array} {|r|r|}\hline group 1 & group 2 & group 3 \\ \hline 9 & 8 & 6 \\ \hline 9 & 8 & 36 \\ \hline 8 & 7 & 10 \\ \hline 11 & 0 & 10 \\ \hline 18 & 0 & 6 \\ \hline 4 & 4 & \\ \hline 0 & & \\ \hline 0 & & \\ \hline 8 & & \\ \hline 12 & & \\ \hline 12 & & \\ \hline \end{array}

I'm interested in determining if there is a meaningful difference between the means of these three groups. The observations are all independent, both within and between groups. Each observation can take on a value of $0-36$. I plotted the density functions for each group of observations, and (as is to be expected for discrete data with relatively few observations per group), they do not appear to be normally distributed and neither do their residuals. So I cannot simply apply one-way ANOVA.

I had a tough time trying to find a discrete parametric model under which I could properly fit the data. So, I figured that perhaps a permutation test using the F-statistic may provide me with a reasonable approach to determining if the means of any one of these 3 groups are significantly different. upon plotting the distribution of my test statistic after 5000 itterations, it closely resembled the theoretical F distribution for $df_1=2,df_2=19$.

To my understanding, the p-value I obtained by running this permutation test represents the probability of obtaining an observed test statistic as large as my initial test statistic (ie. the F-stat calculated from my original data set) assuming the null hypothesis to be true. Given that my p-value was somewhat large, $0.131$, I can conclude that there isn't significant evidence against the null hypothesis and that the means between these 3 groups do not differ significantly.

Is my interpretation of this permutation test correct?

  • 1
    $\begingroup$ I am not familiar with the approach you have used. I understand a permutation test to be a simple nonparametric statistical test, well explained in this interactive visualisation: jwilber.me/permutationtest. Perhaps this could prove to be useful in your analysis. $\endgroup$ – ztkpat001 Aug 10 '20 at 18:53
  • 3
    $\begingroup$ +1 I can reproduce the $0.131.$ A permutation test is perfectly applicable here. $\endgroup$ – whuber Aug 10 '20 at 19:04
  • $\begingroup$ @ptz010atk I used this webpage when determining how to run my permutation test: link $\endgroup$ – DTMD422 Aug 10 '20 at 20:53
  • $\begingroup$ @whuber Thank you very much for checking the results and confirming my answer! I really appreciate it. I have a quick additional question for you if you don't mind answering: Suppose I have zero-inflated count data (ie. most of the observations tend to be around zero, but not all). Is there any reason that this permutation test format wouldn't work? I've tried running the tests and the results seem reasonable, but then again, intuition isn't always correct! $\endgroup$ – DTMD422 Aug 10 '20 at 20:57
  • 1
    $\begingroup$ The permutation test might not be very powerful when an appreciable fraction of the counts are zeros. That's not a problem if the result is significant, but if the result is not quite significant, you might regret having chosen such a test. $\endgroup$ – whuber Aug 10 '20 at 21:00

Yes, a permutation test makes sense here.

Strictly speaking, the null hypothesis is that the distributions are the same, not just that they have the same means. (If they had same means but difference variances, the test would have the wrong Type I error rate.)

Here's example code

> df<-data.frame(group=factor(rep(c(1,2,3),c(11,6,5))),
+               count=c(9,9,8,11,18,4,0,0,
+               8,12,12,8,8,7,0,0,4,
+               6,36,10,10,6))
> obsF<-summary(aov(count~group,data=df))[[1]]$"F value"[1]
> Fs<-replicate(50000,{
+ i<-sample(nrow(df))
+ summary(aov(count~group[i],data=df))[[1]]$"F value"[1]
+ })
> qqplot(qf((1:50000)/50001,2, 19),Fs, xlab="Expected",ylab="Simulation")
> abline(0,1)
> mean(Fs>=obsF)
[1] 0.12894

As you see, I get 0.129, which agrees with what you got up to Monte Carlo uncertainty.

A quantile-quantile plot against the $F^2_{19}$ distribution shows good agreement for small values of the statistic, but less good at large values -- it's harder to get very high $F$ statistics than it would be with iid Normal data.

qqplot vs F distribution

The agreement isn't surprising: in two-sample testing the permutation distribution of the (two-tailed) $t$ statistic is close to a $t$ distribution whether or not the assumptions are satisfied, and this is the same sort of thing.

I might have tried ANOVA on the square roots of the counts instead, to try to stabilise the variance-mean relationship at bit. In these data this gives essentially the same $p$-value, though.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.