4
$\begingroup$

I think I know how the permutation test works when computing the non-parameteric empirical p-value to test for the mean difference between groups:

  1. permute the labels for each group in your original data
  2. compute the percentage of simulations where the simulated statistic was more extreme in the direction of the alternative hypothesis test, than observed.
  3. The percentage is the permutation-based p-value.

My question is, if you have severely imbalanced labels for your groups, do you rebalance them when you permute in step #1? Why or why not?

If, for example, I have 30 examples in class A and 4 in class B, when I permute the labels can I rebalance them to have 17 in both class A and class B?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ If you "rebalance" you aren't doing a permutation. $\endgroup$
    – Wayne
    Commented Feb 21, 2016 at 20:16

1 Answer 1

Reset to default
2
$\begingroup$

No, of course, the two groups will have the number of observations like the data: one group with 4 items, the other 30.The total of different pairs is exactly 34C4=34C30= 34! / (30!*4!).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Even though I think your answer is correct, just saying that this is true "of course" is not very helpful. Why is it important to preserve the sizes of the groups? What happens if one doesn't? $\endgroup$
    – amoeba
    Commented Feb 21, 2016 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.