Has anyone dealt with the issue of computing confidence intervals for the correlation coefficient (parametric or non-parametric) and having to include a multiplicity adjustment factor? I was wondering if this can be done using the bootstrap. Any R package for this purpose.
Instead of just adjusting $P$-values when testing a large number of correlation coefficients, one can use the bootstrap to compute confidence intervals for the ranks of all the correlations. For example, the apparently strongest $|r|$ over 100 coefficients may have a 0.95 bootstrap confidence interval for its rank of [1, 20], i.e. the data are consistent with that pair of variables being only the twentieth most strongly related pair as well as being the best.
The nice thing about confidence intervals for ranks is that they are penalized for multiplicity in a data-sensitive way and demonstrate how hard it is to really find 'winners' and 'losers'.