p-value adjustment via permutation for multiple correlations

Question

I have the following problem: I have performed 3200 correlations between a certain measure of connectivity and 1 behavioural outcome in a sample of subjects. As the single correlations (represented by R and p) are not independent from each other, Bonferroni or FDR correction (at given p-values) failed to leave any significant correlations.

So I tried to follow this permuation approach: While permutating the available subject's connectivity data on the one hand, and also permutating the behavioural data on the other , I created a new pseudo-sample (in which subjects connectivity and behaviour are not matching anymore) for which I calculated all correlations again (3200). I did this 10000x and thus generated 10000x3200 random p-values based on the actual data set. Having ordered all p-values (low-to-high) I looked at the lowest 5% of those p-values (alpha): Random correlations would reach a p-value of 0.01 by 5% chance. In another group, the 5% lowest p-value would only reach 0.052.

My question would be whether this would be a plausible way to correct for multiple correlations in a dataset with lots of non-independent single correlations. Particularly, is it reasonable to use an adjusted p-value that is even higher than the original one, 0.05. Maybe the chance to reach 0.05 on that data set is even lower than 5% due to variance etc.

I am looking forward to your answers and comments.

Thank you very much

Best, Robert

Answer 1

Westfall and Young have proposed resampling approaches to p-value adjustent which are applied in SAS using the MULTTEST procedure. Permmutation and bootstrap methods are used but may be different from what you are doing. You can check the SAS users guide for the MULTTEST procedure or read their book. Here is an amazon link to their book. http://www.amazon.com/Resampling-Based-Multiple-Testing-Adjustment-Probability/dp/0471557617/ref=sr_1_1?s=books&ie=UTF8&qid=1340734744&sr=1-1&keywords=Westfall+and+Young

Answer 2

Before trying heavy computations, you might try more powerful methods. I can only assume you used Benjamini-Hochberg for FDR control. There are several "adaptive" methods in the sense they try to estimate the proportion of true null hypotheses, and use that proportion for a tighter FDR control. Either google "adaptive FDR" or have a look at Blanchard's intro video: http://videolectures.net/acs07_blanchard_apc/

Re permutations--I admit I did not fully understand what you tried but here are some thoughts:

1. I can only assume the 3200 correlations are different locations and you want to conserve spatial dependence.
2. Under a global null hypothesis of no correlation anywhere, the assignment of connectivity to behavior is arbitrary, so you can reassign permute either behavior or connectivity over subjects.
3. Keep in mind to permute all locations similarly to conserve spatial dependence.
4. You can control the FWE by assuming the global null, permuting as described and setting the cutoff at the 1-alpha percentile of the distribution of the minimal p-value. Note that your conclusion would be: "there is correlation somewhere".
5. Using permutations for FDR control is more delicate as you will not want to assume a global null, so you cannot permute freely at all locations. I admit I do not see how to go about.