Multiple Comparison Problems

Question

I am conducting a statistical analysis on data that has already been collected pertaining to early life exposures (e.g. timing of introduction of solid food) and risk of disease later in life (e.g. diabetes).

I have a lot of data to work with, so I was advised to include covariates that have been associated with the outcome. However, i was told that there may be chance results due to multiple comparisons and to use bonferroni or some other mechanism to account for this. Can anyone explain what exactly the problem of "multiple comparison" really means and what accounting for it essentially means?

Answer 1

Have you seen the wikipedia article?

https://en.wikipedia.org/wiki/Multiple_comparisons_problem

Basically, you need it to correct for false positive rates. In the Wikipedia article, there is a section:

However, for 100 tests where all null hypotheses are true ... These errors are called false positives ...

Bonferroni is the simplest mechanism but not recommended if you have many tests due to low statistical power. Other possibilities include Tukey and Dunnett for controlling the familywise error rate. FDR is another possibility.

Answer 2

The book by Peter Westfall and S. Stanley Young was published by Wiley in 1993. The title is Resampling-Based Multiple Testing Examples and Methods for p-value Adjustment.

This covers the permutation and bootstrap tests and includes their software implementation, Proc Multtest in SAS.

Multiple Comparisons is a very general and necessary procedure to avoid false positives check out the work of Yoav Benjamini who has done a great deal of research on the use of false discovery rate as a measure.

There has been a lot of discussion on this site recently questioning whether or not multiple comparisons are needed when various statistical tests are planned in advance.

Answer 3

Here is how things might go wrong with covariates: Say you have a main treatment variable of interest, maybe a dummy variable. You are really interesting in significance of that variable. Now, say you have 10 covariates, each of which can either be in out of your model, leading to 2^10 = 1024 possible models, each of which also contains your treatment variable. Each model gives a different p-value for your treatment variable, so now you have 1024 p-values. An unscrupulous or ignorant researcher might take the smallest of those 1024 p-values, and call that the "evidence" for the treatment effect.

It should be clear that this smallest p-value out of the 1024 is too small, leading to an inflated type I error rate.

Even if you don't think you are looking at that many p-values, in reality you probably are because I would imagine that your are using some automatic variable screening analysis that looks at all those models implicitly.

With 20 covariates there would be over a million p-values, and the problem would be much worse.

The Bonferroni solution would be to use .05/1024 instead of .05 to determine significance of the treatment variable in the first case, and .05/1,000,000 in the second. Seems a little extreme.

Thanks Michael for the nods. I guess I could envision a resampling solution here, but it's a little nonstandard. Certainly, you'd want to take care of the extremely high dependencies among the p-values, and resampling would do that whereas Bonferroni would fail (as mentioned above, it's pretty darn conservative). The applications in our book were for more standard multiple comparisons in ANOVA or MANOVA. We did deal with covariates, but we assumed they were pre-selected, not either in our out of the model, and thus not the source of the multiple comparisons problem, as is the case here with the original question.

Current research by Brad Efron and others on "post-selection inference" might be best way to go here.