# Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?

I have been calculating p-values for correlations using bootstrapping. I've just been using the same p-value cutoff regardless of the number of pair-wise calculations. Intuitively, it seems to me that the more correlations that I perform, the more likely I am to obtain a spurious correlation due to random chance.

Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?

-
Can you explain this a little more clearly? Are you calculating correlation estimates between two variable that are paired? What is your null hypothesis? What bootstrap distribution are you using under the null hypothesis? It is definitely true that the more variables you test the more likely you are to commit a type 1 error unless you adjust for multiplicity. –  Michael Chernick Jun 18 '12 at 19:15
I'm assuming the variables aren't paired. The data is counts of species drawn across multiple (but independent) sample sites, with a null hypothesis that there is no correlation between population sizes of species across the environment at the sample sites. I'm also not making any assumptions about the distribution in bootstrapping. –  zzk Jun 19 '12 at 20:44
If you don't pair the Xs and Ys what do you mean by correlation? Seems that you must pairing the species population size with some environmental factor at the site. I know that the bootstrap distirbution is gotten by resampling and does not have any assumed form. What I meant was are you using the null hypothesis to center the test statistic based on the bootstrap distribution which is what Hall and Wilson recommend. Also did you know that Westfall and Young developed around 1993 a method for p-value adjustment due to multiple testing. –  Michael Chernick Jun 19 '12 at 20:57
You can find out more about that in their book. Hers is a link to it on amazon: amazon.com/… –  Michael Chernick Jun 19 '12 at 20:58
Here are links to the Hall and Wilson paper:jstor.org/discover/10.2307/… and mendeley.com/research/… –  Michael Chernick Jun 19 '12 at 21:03
show 1 more comment

I also don't follow your situation 100%, but I suspect it doesn't matter. The problem of multiple comparisons arises simply due to the mathematics of looking at lots of random things. That is, each statistical test can be understood as a Bernoulli trial. If the null hypothesis holds in every case, you have a Binomial distribution with probability .05 and N equal to the number of tests. (If the null never holds, you have a binomial with probability equal to the statistical power and the same N.) Thus, if the null is always true, and the tests are independent, the probability of not making any type I errors is $.95^N$.