Right way for permutation test for correlation

Question

I have a data matrix with size of 24 by 369, consisting of 4 classes. I want to evaluate the variable importance using permutation test. I know there are lots of methods to find informative variables according to the question at hand, but here I would like to focus on correlation between variable x and class y, as for multiple class problems, this is also an effective way to evaluate the importance of variables. Two ways for permutation test are used:

1. Randomly shuffle y and calculate the correlation between shuffled y and x. Repeat this for 10, 000 times and calculate the fraction of correlations larger than correlation between normal y and x (denoted as normal correlation) as the estimated p value. Than use Benjamini & Hochberg correction procedure to get the variables with p values lower then threshold defined by FDR of 5%, as a multiple comparison manner.
2. For all 369 variables, the largest correlation in correlations calculated from shuffled y and each variable x, as in way 1, is collected (denoted as null correlation). Thus for 369 variables I have 369 null correlations, sorting in ascending order. Then find the position of each normal correlation in null correlations. Select the variables with normal correlation in top 5% as a control of FDR 5%.

I can get several variables from way 1, but none from way 2. Am I doing anything wrong, especially in way 2 as it seems to be also a popular way for multiple comparison? Further question is, what is the difference between these two ways?

Answer 1

The difference is in the multiplicity measure you are controlling. Benjamini-Hochberg procedure controls FDR, while the second one you are using - permutation max-T procedure - controls FWER. So, on one hand, the second procedure takes into account the structure of correlations between test statistics, which makes it more powerful (able to detect true deviations from the null); on the other, it enforces stricter control for multiplicity, which makes it less powerful. It seems that in your case the second force is stronger.

By the way, are you using Pearson correlations? Since $y$ is a 4-valued discrete variable, I would suggest replacing correlations with, for example, one-way ANOVA Fisher's test statistics: $$ F = \frac{\frac1{4-1} \sum\limits_{k=1}^4 n_k \left(\bar{x}_k - \bar{x}\right)^2}{\frac1{N-4}\sum\limits_{k=1}^4\sum\limits_{i=1}^{n_k} \left(x_{ki} - \bar{x}_k\right)^2 },$$ where $\bar{x}_k$'s are averages $x$ values for the $k$'th level of $y$, $\bar{x}$ is the global average, and $n_k$ is the number of cases with $y=k$. This statistic measures the difference between average values of $x$ for different levels of $y$ without imposing a (possibly nonexistent) order relation between different class labels.