I think I know how the permutation test works when computing the non-parameteric empirical p-value to test for the mean difference between groups:
- permute the labels for each group in your original data
- compute the percentage of simulations where the simulated statistic was more extreme in the direction of the alternative hypothesis test, than observed.
- The percentage is the permutation-based p-value.
My question is, if you have severely imbalanced labels for your groups, do you rebalance them when you permute in step #1? Why or why not?
If, for example, I have 30 examples in class A and 4 in class B, when I permute the labels can I rebalance them to have 17 in both class A and class B?