I think I know how the permutation test works when computing the non-parameteric empirical p-value to test for the mean difference between groups:

  1. permute the labels for each group in your original data
  2. compute the percentage of simulations where the simulated statistic was more extreme in the direction of the alternative hypothesis test, than observed.
  3. The percentage is the permutation-based p-value.

My question is, if you have severely imbalanced labels for your groups, do you rebalance them when you permute in step #1? Why or why not?

If, for example, I have 30 examples in class A and 4 in class B, when I permute the labels can I rebalance them to have 17 in both class A and class B?

  • 2
    $\begingroup$ If you "rebalance" you aren't doing a permutation. $\endgroup$
    – Wayne
    Feb 21, 2016 at 20:16

1 Answer 1


No, of course, the two groups will have the number of observations like the data: one group with 4 items, the other 30.The total of different pairs is exactly 34C4=34C30= 34! / (30!*4!).

  • 1
    $\begingroup$ Even though I think your answer is correct, just saying that this is true "of course" is not very helpful. Why is it important to preserve the sizes of the groups? What happens if one doesn't? $\endgroup$
    – amoeba
    Feb 21, 2016 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.