I was trying to do a permutation test on a large amount of temperature observations in R. Approximately 1700 temperature observations (g1) from one place and 2 times 1700 observations from two other places (g2).

I would like to test the hull hypothesis $\mu_1-\mu_2 = 0$ against the alternative hypothesis $\mu_1-\mu_2 < 0$. I am doing the following calculations:

k            <– 10000
perms        <- replicate(k, sample(c(g1, g2)))
my.mean.difs <- apply(perms[1:length(g1), ], 2, mean) -
                apply(perms[(length(g1)+1):(length(g1)+length(g2)), ], 2, mean)

p            <- sum(my.mean.difs < (mean(g1) - mean(g2))) / k

The p-value, p, gives me 0, straight. I don't understand why I am not getting any difference in the means (my.mean.difs) which is smaller than the difference in the actual mean. My immediate guess is that I am using too few resamples, but since the data sets are so big (1700 and 3400 data points) it will be very time consuming to do many more resamples. I could randomly create some smaller data sets, and test them, but I am not sure that is a good procedure.

Can anybody explain me why I am getting a p-value of 0, straight, and if it is a bad idea to do permutation on big data sets like this?

Bonus info:

Temperatures in group 1 (g1)

  • $\mu_1 = 24.14$ °C
  • $\sigma_1 = 0.96$

Temperatures in group 2 (g2)

  • $\mu_2 = 24.95$ °C
  • $\sigma_2 = 1.01$


Based on @whuber's comment and the central limit theorem I tried to make a two-sample Z-test of the means, which should be OK to use here, if the assumption of equal variances is true, right? I did the following calculations:

(mean(g1)-mean(g2)) / sqrt(var(g1)/length(g1) + var(g2)/length(g2))

This gives me an extremely small Z-value of -28.25. Is this magnitude of the Z-value realistic?

Sorry for the probably very basic question, but I am quite new to statistics, and from my intro stat course, we worked on much smaller sample sizes, and Z-values much closer to 0. But I guess that the large sample sizes makes it possible to show a clearly significant difference in the means, even if the actual difference not is that big. It that right?

  • 2
    $\begingroup$ If the groups were approximately normally distributed, your chance of observing a smaller difference of means within the permutation data would be approximately $10^{-171}$. Even with highly skewed groups the chance is going to be pretty small! $\endgroup$ – whuber Sep 2 '14 at 21:42
  • $\begingroup$ Your edit answers the original question pretty nicely :-). $\endgroup$ – whuber Sep 3 '14 at 17:05
  • $\begingroup$ @Christoffer - you could turn your edit into an answer. $\endgroup$ – Glen_b Sep 6 '14 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.