|
I have been calculating p-values for correlations using bootstrapping. I've just been using the same p-value cutoff regardless of the number of pair-wise calculations. Intuitively, it seems to me that the more correlations that I perform, the more likely I am to obtain a spurious correlation due to random chance. Does using bootstrapping change how you deal with problems of Type I errors when testing multiple correlations? |
|||||||||||
|
|
I also don't follow your situation 100%, but I suspect it doesn't matter. The problem of multiple comparisons arises simply due to the mathematics of looking at lots of random things. That is, each statistical test can be understood as a Bernoulli trial. If the null hypothesis holds in every case, you have a Binomial distribution with probability .05 and N equal to the number of tests. (If the null never holds, you have a binomial with probability equal to the statistical power and the same N.) Thus, if the null is always true, and the tests are independent, the probability of not making any type I errors is $.95^N$. Bootstrapping does not get you out of this fact. Bootstrapping offers a way to deal with situations in which your test statistic may not follow the distribution assumed by large sample theory. (This can occur because the distribution of the data is too non-normal, and your sample isn't large enough to compensate; n.b. in some cases, e.g. Cauchy data, a sample can never be large enough.) Provided your data are representative of the population in question, Bootstrapping may allow you to calculate an appropriate p-value (some conditions apply). However, this issue is orthogonal to the problem of multiple comparisons; that is, bootstrapping would give you the appropriate p-value for a 'family' of size 1. The problem of multiple comparisons is typically discussed in terms of multiple t-tests. I gather you are clear about the fact that using correlations instead of t-tests is irrelevant. Using bootstrapped sampling distributions instead of analytical sampling distributions is completely analogous in this respect. Having made these points, the question arises of what to do about the problem of multiple comparisons in your case, given that bootstrapping is not offering you any protection. You should know that this topic has long been somewhat controversial, with scholars debating different strategies and even whether it's worthwhile to bother with the issue. There is a good deal of discussion about multiple comparisons on CV; if you search on the tag (i.e., click on it) you will be able to get a lot of information. |
|||
|
|
