Correlation analysis and correcting p-values for multiple testing II

Question

I am currently facing problems dealing with lots of individual simple correlations (28 individual DTI connections strengths, individual behavioural scores on 31 time points). That would be 28x31 individual correlations. Referring to a previous discussion on that, I just want to make sure that I understood correctly and am looking forward for your short answers whether this way would be fine:

1. Individual correlations 28x31 are computed, giving R and p values (original data).
2. Subjects are shuffled on 1 side (e.g. for all 28 connections, pseudosample), all correlations are computed again, giving a new 28x31x5000 matrix with R and p-val. 2a. This is done 5000x.
3. The number R perm > R obs (N) is collected for each correlation (hence max. 5000).
4. Emperical p val is computed as N/5000 for each correlation.
5. A common threshold of p-val of e.g. 0.05 is applied to find significant correlations based on all individual 28x31 p-vals.

My additional questions would be:

A. What about negative R? What about the case when abs(R) would be larger in perm. than in original?
B. How to adress R perm -0.05 and R original -0.6, hence stronger negative correlation? At this stage, I only count the # of Rorig > Rperm, should I also count abs(Rorig> abs (Rperm)?

I am still a bit confused and unsure whether I would report correct statistics.

Answer 1

This may be more of a selection problem than a hypothesis testing problem. If that is the case you would be better served by using the bootstrap to obtain 0.95 confidence intervals on the ranks of all the correlation coefficients (or ranks of their squares or absolute values). In doing the bootstrap, keep all rows of the data matrix together, i.e., don't use random permutations of any of the variables. This bootstrap analysis automatically adjusts for multiple comparisons, and exposes the true difficulty of the task by the width of the confidence intervals.

Answer 2

If you have negative correlations than logically you should change the sign $>$ to $<$ on step 3 for this particular comparison. To this end use $\left|\frac{R_{\text{perm}}}{R_{\text{obs}}}\right|>1$ rule in step 3.