# Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?

I have been calculating p-values for correlations using bootstrapping. I've just been using the same p-value cutoff regardless of the number of pair-wise calculations. Intuitively, it seems to me that the more correlations that I perform, the more likely I am to obtain a spurious correlation due to random chance.

Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations?

-
 Can you explain this a little more clearly? Are you calculating correlation estimates between two variable that are paired? What is your null hypothesis? What bootstrap distribution are you using under the null hypothesis? It is definitely true that the more variables you test the more likely you are to commit a type 1 error unless you adjust for multiplicity. – Michael Chernick Jun 18 '12 at 19:15 I'm assuming the variables aren't paired. The data is counts of species drawn across multiple (but independent) sample sites, with a null hypothesis that there is no correlation between population sizes of species across the environment at the sample sites. I'm also not making any assumptions about the distribution in bootstrapping. – zzk Jun 19 '12 at 20:44 If you don't pair the Xs and Ys what do you mean by correlation? Seems that you must pairing the species population size with some environmental factor at the site. I know that the bootstrap distirbution is gotten by resampling and does not have any assumed form. What I meant was are you using the null hypothesis to center the test statistic based on the bootstrap distribution which is what Hall and Wilson recommend. Also did you know that Westfall and Young developed around 1993 a method for p-value adjustment due to multiple testing. – Michael Chernick Jun 19 '12 at 20:57 You can find out more about that in their book. Hers is a link to it on amazon: amazon.com/… – Michael Chernick Jun 19 '12 at 20:58 Here are links to the Hall and Wilson paper:jstor.org/discover/10.2307/… and mendeley.com/research/… – Michael Chernick Jun 19 '12 at 21:03
show 1 more comment

## 1 Answer

I also don't follow your situation 100%, but I suspect it doesn't matter. The problem of multiple comparisons arises simply due to the mathematics of looking at lots of random things. That is, each statistical test can be understood as a Bernoulli trial. If the null hypothesis holds in every case, you have a Binomial distribution with probability .05 and N equal to the number of tests. (If the null never holds, you have a binomial with probability equal to the statistical power and the same N.) Thus, if the null is always true, and the tests are independent, the probability of not making any type I errors is $.95^N$.

Bootstrapping does not get you out of this fact. Bootstrapping offers a way to deal with situations in which your test statistic may not follow the distribution assumed by large sample theory. (This can occur because the distribution of the data is too non-normal, and your sample isn't large enough to compensate; n.b. in some cases, e.g. Cauchy data, a sample can never be large enough.) Provided your data are representative of the population in question, Bootstrapping may allow you to calculate an appropriate p-value (some conditions apply). However, this issue is orthogonal to the problem of multiple comparisons; that is, bootstrapping would give you the appropriate p-value for a 'family' of size 1.

The problem of multiple comparisons is typically discussed in terms of multiple t-tests. I gather you are clear about the fact that using correlations instead of t-tests is irrelevant. Using bootstrapped sampling distributions instead of analytical sampling distributions is completely analogous in this respect.

Having made these points, the question arises of what to do about the problem of multiple comparisons in your case, given that bootstrapping is not offering you any protection. You should know that this topic has long been somewhat controversial, with scholars debating different strategies and even whether it's worthwhile to bother with the issue. There is a good deal of discussion about multiple comparisons on CV; if you search on the tag (i.e., click on it) you will be able to get a lot of information.

-