3
$\begingroup$

I am trying to understand properly the significance tests using permutation. I am wondering if this code is correct. Let us say I want to test the difference in mean for mpg by vs 

library(dplyr) 
library(mtcars) 

mtcars %>% summarise(meandiff = mean( mpg[vs == 1] - mpg[vs == 0] ))
# 7.3

I was thinking of doing the simulation this way 

tot = 1:20
vt  = vector('list', 1000)
for(i in 1:1000){
  num1 = sample(tot, 13)
  num2 = tot[!(tot %in% num1)] 
  vt[[i]] = mtcars$mpg[num1] %>% mean() - mtcars$mpg[num2] %>% mean()
}
unlist(vt) %>% hist(breaks = 100)

enter image description here

So now I want to determine the probability that the difference in mean of 7 occurred by chance only.

Can I use the normal distribution command here? 

qnorm(0.95, mean=mean(vta), sd=sd(vta))
# 5

So can I conclude with $\alpha = 5\%$ that a difference in mean of 7 is statistically significant?

From a centile point of view, the $\alpha = 5\%$ can be visualised like this: 

vtas = vta %>% sort()
vtas %>% plot
abline(h = vtas[950])

enter image description here

  • Is this a correct way to do and interpret this result?
  • It seems to me that the permutation test requires less assumptions than other types of test. Is that right?
Improve this question
$\endgroup$
7
  • 3
    $\begingroup$ Code check questions are off topic here--they belong on Code Review. That said, I don't think this is really code check, I think this is a question about understanding the statistical ideas here. As such, it would be on topic here, but it is worth being clear about your actual question, lest this thread be closed by mistake. $\endgroup$
    – gung - Reinstate Monica
    Dec 15, 2015 at 19:25
  • 1
    $\begingroup$ And as not everyone uses R, commenting code is helpful. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Dec 15, 2015 at 19:51
  • 1
    $\begingroup$ @gung. I understand. I think this post is about general understanding and application of the method. I am simply looking for some comments on the code and on my interpretation. I am not sure where does a post like that would fit. thanks $\endgroup$
    – giac
    Dec 15, 2015 at 19:52
  • $\begingroup$ +1. Very good questions, especially the one about normal distribution. Thanks @gung for the edit. $\endgroup$
    – amoeba
    Dec 15, 2015 at 20:03
  • 1
    $\begingroup$ @giacomoV The histogram gives an estimated distribution for the mean difference when vs has no effect, so you can use this directly to determine a p-value. If we believe that the normal approximation is a good one, then we might as well just use a t-test instead of a permutation test. $\endgroup$
    – dsaxton
    Dec 16, 2015 at 14:22

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.