I am analyzing psych experiments which generally take the form of a list of outcome measures for the test and control groups, with roughly equal sample sizes, and I would like to compare their means using a permutation test. Most answers suggest this is OK, but this article suggests that Type I errors will be inflated: https://academic.oup.com/bioinformatics/article/22/18/2244/317881.

My understanding from the paper above is that if I am testing for a difference in means using a permutation test, I may get a Type I error/false positive in a situation where the means are equal, but some other feature of the data like variance or skewness differs. In normal life, variance/skewness of data is basically never equal, so I'm having trouble getting an intuitive sense of what this should mean in practice.

Given this, when and how should I conclude that a permutation test is appropriate for testing a difference in means, if ever?

  • $\begingroup$ If the null hypothesis is true, do you have any reason to believe the outcome measures would have different distributions? $\endgroup$ – jbowman Feb 8 '18 at 1:46
  • $\begingroup$ My concern is that a treatment could do something like increase variance without changing the mean. For example, If I’m analyzing how changes in question framing alter survey responses on a Likert scale, I could imagine a question might make people have stronger feelings about the subject but not change the direction of their feelings/the mean. then my permutation test might be incorrect. Maybe I’m overthinking this? $\endgroup$ – verybadatthis Feb 8 '18 at 2:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.