I think I know how the permutation test works when computing the non-parameteric empirical p-value to test for the mean difference between groups:

  1. permute the labels for each group in your original data
  2. compute the percentage of simulations where the simulated statistic was more extreme in the direction of the alternative hypothesis test, than observed.
  3. The percentage is the permutation-based p-value.

My question is, if you have severely imbalanced labels for your groups, do you rebalance them when you permute in step #1? Why or why not?

If, for example, I have 30 examples in class A and 4 in class B, when I permute the labels can I rebalance them to have 17 in both class A and class B?

  • 1
    $\begingroup$ If you "rebalance" you aren't doing a permutation. $\endgroup$ – Wayne Feb 21 '16 at 20:16

No, of course, the two groups will have the number of observations like the data: one group with 4 items, the other 30.The total of different pairs is exactly 34C4=34C30= 34! / (30!*4!).

  • 1
    $\begingroup$ Even though I think your answer is correct, just saying that this is true "of course" is not very helpful. Why is it important to preserve the sizes of the groups? What happens if one doesn't? $\endgroup$ – amoeba says Reinstate Monica Feb 21 '16 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.