I have a data matrix with size of 24 by 369, consisting of 4 classes. I want to evaluate the variable importance using permutation test. I know there are lots of methods to find informative variables according to the question at hand, but here I would like to focus on correlation between variable
x and class
y, as for multiple class problems, this is also an effective way to evaluate the importance of variables. Two ways for permutation test are used:
- Randomly shuffle
yand calculate the correlation between shuffled
x. Repeat this for 10, 000 times and calculate the fraction of correlations larger than correlation between normal
x(denoted as normal correlation) as the estimated p value. Than use Benjamini & Hochberg correction procedure to get the variables with p values lower then threshold defined by FDR of 5%, as a multiple comparison manner.
- For all 369 variables, the largest correlation in correlations calculated from shuffled
yand each variable
x, as in way 1, is collected (denoted as null correlation). Thus for 369 variables I have 369 null correlations, sorting in ascending order. Then find the position of each normal correlation in null correlations. Select the variables with normal correlation in top 5% as a control of FDR 5%.
I can get several variables from way 1, but none from way 2. Am I doing anything wrong, especially in way 2 as it seems to be also a popular way for multiple comparison? Further question is, what is the difference between these two ways?