# Correlation analysis and correcting $p$-values for multiple testing

I am investigating the effect of a drug on EEG and cognition in epilepsy patients. We have done EEG and neuropsychological (NP) tests twice, before and after medical treatment. Some of parameters in EEG and NP tests were significantly altered after drug treatment (done by Wilcoxon rank sum test, as the distribution was not normal).

Now, I would like to calculate correlation between change in EEG parameters and change in NP tests. There are 7 parameters in EEG parameters and 27 in NP tests. I would like to run correlation analysis on every pair and see if any pair is significantly correlated.

My questions are:

1. Should I use nonparametric correlation, say Spearman, to calculate $\rho$? (as values of EEG and NP paramaters are not normal)

2. Since there are 7 parameters in EEG and 27 parameters in NP tests, there should be 7*27 number of testing, which requires correction of p value for multiple comparisons. Which method should I use?

I am looking for not-so-conservative method. I have seen from another posting that I can use permutation test for multiple comparisons. Could someone explain how I can do this?

-
About the permutation approach, see @whuber's answer to this related question, Look and you shall find (a correlation), which describes the basic idea of breaking the structure or link between your two series of measurements. –  chl Mar 15 '11 at 9:55

In my opinion you should test if your variables are distributed normally and chose a suitable test accordingly.

Concerning the correction for alpha inflation: What you are doing is data mining. You have experimental data and now you are digging in it to find ... anything. Do that. But also know that anything you might find is just an observation and as such not reliable. Perform that exploratory thing, pick some promising pairs of variables and conduct new experiments to test these pairs for correlation.

-
This seems a little harsh. It sounds like the search for correlations was planned at the outset and the OP is aware of the multiple comparisons problem. That already puts them on a better experimental, analytical, and conceptual footing than the vast majority of publications in this field. –  whuber Mar 15 '11 at 15:25
You want to perform a canonical correlation analysis. This will provide information about correlations among linear combinations of your sets of parameters, potentially uncovering stronger information of the type you seek. The Wikipedia article explains the theory, provides the equations, and presents the appropriate hypothesis test. It requires solving a 7 by 7 eigenvalue problem determined by the correlation and cross-correlation matrices of the variables, which is fast and straightforward. The R package CCorA will do it, for instance. (I haven't tested this package.)
@Ryan (If you want to address @whuber directly, don't forget to prefix its username with @ so that he can be notified.) I don't really like to intervene like this, but what is your "original" correlation about? I mean, a perm. test will inform you about the prob. of observing such a stat. ($r$ in your case) at least as extreme as the one you observed in your data (and with 10,000 draws, this is largely enough to conclude at a 5% level); then reporting your $p$-value is just a matter of computing the No. times you observed $r>r_\text{obs}$, where $r_\text{obs}$ = your empirical corr. –  chl Mar 16 '11 at 20:41