I am analyzing psych experiments which generally take the form of a list of outcome measures for the test and control groups, with roughly equal sample sizes, and I would like to compare their means using a permutation test. Most answers suggest this is OK, but this article suggests that Type I errors will be inflated: https://academic.oup.com/bioinformatics/article/22/18/2244/317881.

My understanding from the paper above is that if I am testing for a difference in means using a permutation test, I may get a Type I error/false positive in a situation where the means are equal, but some other feature of the data like variance or skewness differs. In normal life, variance/skewness of data is basically never equal, so I'm having trouble getting an intuitive sense of what this should mean in practice.

Given this, when and how should I conclude that a permutation test is appropriate for testing a difference in means, if ever?

  • If the null hypothesis is true, do you have any reason to believe the outcome measures would have different distributions? – jbowman Feb 8 at 1:46
  • My concern is that a treatment could do something like increase variance without changing the mean. For example, If I’m analyzing how changes in question framing alter survey responses on a Likert scale, I could imagine a question might make people have stronger feelings about the subject but not change the direction of their feelings/the mean. then my permutation test might be incorrect. Maybe I’m overthinking this? – verybadatthis Feb 8 at 2:28

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.