# Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the service yes/no, and the number of occasions). You have 10 predictor/independent variables that could theoretically explain the demand (e.g., age, sex, income, price, race, etc). Running two separate multiple regressions will yield 20 coefficients estimations and their p-values. With enough independent variables in your regressions you would sooner or later find at least one variable with a statistically significant correlation between the dependent and independent variables.

My question: is it a good idea to correct the p-values for multiple tests if I want to include all independent variables in the regression? Any references to prior work are much appreciated.

• hmmm... interesting problem - bivariate regression with one variable [sort of] continuous and one dichotomous. usual MVN regression theory says doing separate regressions on each of two bivariate normal responses - and then piecing the results together - is the correct analysis - in the gauss-markov sense of minimizing the v-c matrix of the regression estimators among all unbiased linear estimators - and in the MVN case, among all unbiased estimators]. is 'separate regressions' still the best one can do when one regression is logistic? [unbiasedness seems a bit of a stretch in that case, excep – ronaf Oct 1 '10 at 4:36

It seems your question more generally addresses the problem of identifying good predictors. In this case, you should consider using some kind of penalized regression (methods dealing with variable or feature selection are relevant too), with e.g. L1, L2 (or a combination thereof, the so-called elasticnet) penalties (look for related questions on this site, or the R penalized and elasticnet package, among others).

Now, about correcting p-values for your regression coefficients (or equivalently your partial correlation coefficients) to protect against over-optimism (e.g. with Bonferroni or, better, step-down methods), it seems this would only be relevant if you are considering one model and seek those predictors that contribute a significant part of explained variance, that is if you don't perform model selection (with stepwise selection, or hierarchical testing). This article may be a good start: Bonferroni Adjustments in Tests for Regression Coefficients. Be aware that such correction won't protect you against multicollinearity issue, which affects the reported p-values.

Given your data, I would recommend using some kind of iterative model selection techniques. In R for instance, the stepAIC function allows to perform stepwise model selection by exact AIC. You can also estimate the relative importance of your predictors based on their contribution to $R^2$ using boostrap (see the relaimpo package). I think that reporting effect size measure or % of explained variance are more informative than p-value, especially in a confirmatory model.

It should be noted that stepwise approaches have also their drawbacks (e.g., Wald tests are not adapted to conditional hypothesis as induced by the stepwise procedure), or as indicated by Frank Harrell on R mailing, "stepwise variable selection based on AIC has all the problems of stepwise variable selection based on P-values. AIC is just a restatement of the P-Value" (but AIC remains useful if the set of predictors is already defined); a related question -- Is a variable significant in a linear regression model? -- raised interesting comments (@Rob, among others) about the use of AIC for variable selection. I append a couple of references at the end (including papers kindly provided by @Stephan); there is also a lot of other references on P.Mean.

Frank Harrell authored a book on Regression Modeling Strategy which includes a lot of discussion and advices around this problem (§4.3, pp. 56-60). He also developed efficient R routines to deal with generalized linear models (See the Design or rms packages). So, I think you definitely have to take a look at it (his handouts are available on his homepage).

References

1. Whittingham, MJ, Stephens, P, Bradbury, RB, and Freckleton, RP (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75, 1182-1189.
2. Austin, PC (2008). Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study. Journal of Clinical Epidemiology, 61(10), 1009-1017.
3. Austin, PC and Tu, JV (2004). Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. Journal of Clinical Epidemiology, 57, 1138–1146.
4. Greenland, S (1994). Hierarchical regression for epidemiologic analyses of multiple exposures. Environmental Health Perspectives, 102(Suppl 8), 33–39.
5. Greenland, S (2008). Multiple comparisons and association selection in general epidemiology. International Journal of Epidemiology, 37(3), 430-434.
6. Beyene, J, Atenafu, EG, Hamid, JS, To, T, and Sung L (2009). Determining relative importance of variables in developing and validating predictive models. BMC Medical Research Methodology, 9, 64.
7. Bursac, Z, Gauss, CH, Williams, DK, and Hosmer, DW (2008). Purposeful selection of variables in logistic regression. Source Code for Biology and Medicine, 3, 17.
8. Brombin, C, Finos, L, and Salmaso, L (2007). Adjusting stepwise p-values in generalized linear models. International Conference on Multiple Comparison Procedures. -- see step.adj() in the R someMTP package.
9. Wiegand, RE (2010). Performance of using multiple stepwise algorithms for variable selection. Statistics in Medicine, 29(15), 1647–1659.
10. Moons KG, Donders AR, Steyerberg EW, and Harrell FE (2004). Penalized Maximum Likelihood Estimation to predict binary outcomes. Journal of Clinical Epidemiology, 57(12), 1262–1270.
11. Tibshirani, R (1996). Regression shrinkage and selection via the lasso. Journal of The Royal Statistical Society B, 58(1), 267–288.
12. Efron, B, Hastie, T, Johnstone, I, and Tibshirani, R (2004). Least Angle Regression. Annals of Statistics, 32(2), 407-499.
13. Flom, PL and Cassell, DL (2007). Stopping Stepwise: Why stepwise and similar selection methods are bad, and what you should use. NESUG 2007 Proceedings.
14. Shtatland, E.S., Cain, E., and Barton, M.B. (2001). The perils of stepwise logistic regression and how to escape them using information criteria and the Output Delivery System. SUGI 26 Proceedings (pp. 222–226).
• I second chl's recommendation to use some kind of penalized regression (e.g., the Lasso). – Stephan Kolassa Oct 1 '10 at 9:28
• @chl: I'm unhappy with recommending stepwise predictor selection. Usually, this is based on p-values ("exclude a predictor with p>.15, include it if p<.05") and leads to biased estimates and bad predictive performance (Whittingham et al., 2006, Why do we still use stepwise modelling in ecology and behaviour? J Anim Ecol, 75, 1182-1189). However, AIC-based stepwise approaches have the same weakness - Frank Harrell discussed this in a post to R-help on Mon, 09 Aug 2010 16:34:19 -0500 (CDT) in the thread "Logistic Regression in R (SAS -like output)". – Stephan Kolassa Oct 1 '10 at 9:40
• @Stephan +1 Thanks for this. I know Frank Harrell's point of view. His book is a "salvation" for biostatistics, as well as the more recent one by EW Steyerberg. This is why I referred to his work at the end of my response, assuming the interested reader would look for a more thorough explanation. My initial thought was just to point to different ways of approaching the problem. As whuber nicely pointed it out, there's room for improvement in my response, esp. with cross-validation and conceptual issues around predictive modeling. – chl Oct 1 '10 at 9:49
• @Stephan Now the problem is that often we want to keep some variables in our model (whatever their p-values), as @whuber said; this is why I suggested to look at the penalized package from J Goeman, because it allows to penalize only a subset of the covariates. From my experience, I found results from Lasso/Ridge regression a little bit difficult to explain to an external audience, esp. when data don't really called for shrinkage/regularization (e.g. "ideal" case with 500 subjects and 10 variables, no collinearity or mediation issues). So when we can keep things simple without overfitting... – chl Oct 1 '10 at 10:07

To a great degree you can do whatever you like provided you hold out enough data at random to test whatever model you come up with based on the retained data. A 50% split can be a good idea. Yes, you lose some ability to detect relationships, but what you gain is enormous; namely, the ability to replicate your work before it is published. No matter how sophisticated the statistical techniques you bring to bear, you will be shocked at how many "significant" predictors wind up being entirely useless when applied to the confirmation data.

Bear in mind, too, that "relevant" for prediction means more than a low p-value. That, after all, only means it's likely a relationship found in this particular dataset is not due to chance. For prediction it's actually more important to find the variables that exert substantial influence on the predictand (without over-fitting the model); that is, to find the variables that are likely to be "real" and, when varied throughout a reasonable range of values (not just the values that might occur in your sample!), cause the predictand to vary appreciably. When you have hold-out data to confirm a model, you can be more comfortable provisionally retaining marginally "significant" variables that might not have low p-values.

For these reasons (and building on chl's fine answer), although I have found stepwise models, AIC comparisons, and Bonferroni corrections quite useful (especially with hundreds or thousands of possible predictors in play), these should not be the sole determinants of which variables enter your model. Do not lose sight of the guidance afforded by theory, either: variables having strong theoretical justification to be in a model usually should be kept in, even when they are not significant, provided they do not create ill-conditioned equations (e.g., collinearity).

NB: After you have settled on a model and confirmed its usefulness with the hold-out data, it's fine to recombine the retained data with the hold-out data for final estimation. Thus, nothing is lost in terms of the precision with which you can estimate model coefficients.

• Thanks! If you do not have access to the original data but only a table of regression coefficients, is the Bonferroni adjustment your only choice? – Mikael M Sep 30 '10 at 19:08
• Presumably you also have the p-values :-). But with only those and the coefficients, it's hard to imagine what else you might do besides a Bonferroni adjustment. (I always make such an adjustment whenever reading any paper with multiple tests: it's a quick way to winnow out the results that are likely to be junk.) Most people also provide summary statistics for the variables: you can use ranges or sds along with the coefficients to estimate how much effect each explanatory variable might have on the predictand. – whuber Sep 30 '10 at 19:17
• Thanks for your explanation, esp. on cross-validation. I appreciate your last argument, i.e. that we also have to look for theoretical relevance (beyond p-values). – chl Sep 30 '10 at 19:20

I think this is a very good question; it gets to the heart of the contentious multiple testing "problem" that plagues fields ranging from epidemiology to econometrics. After all, how can we know if the significance we find is spurious or not? How true is our multivariable model?

In terms of technical approaches to offset the likelihood of publishing noise variables, I would heartily agree with 'whuber' that using part of your sample as training data and the rest as test data is a good idea. This is an approach that gets discussed in the technical literature, so if you take the time you can probably find out some good guidelines for when and how to use it.

But to strike more directly at the philosophy of multiple testing, I suggest you read the articles I reference below, some of which support the position that adjustment for multiple testing is often harmful (costs power), unnecessary, and may even be a logical fallacy. I for one do not automatically accept the claim that our ability to investigate one potential predictor is inexorably reduced by the investigation of another. The family-wise Type 1 error rate may increase as we include more predictors in a given model, but so long as we do not go beyond the limits of our sample size, the probability of Type 1 error for each individual predictor is constant; and controlling for family-wise error does not illuminate which specific variable is noise and which is not. Of course, there are cogent counter-arguments as well.

So, as long as you limit your list of potential variables to those which are plausible (ie, would have known pathways to the outcome) then the risk of spuriousness is already handled fairly well.

However, I would add that a predictive model is not as concerned with the "truth-value" of its predictors as a causal model; there may be a great deal of confounding in the model, but so long as we explain a large degree of the variance then we aren't too concerned. This makes the job easier, at least in one sense.

Cheers,

Brenden, Biostatistical Consultant

PS: you may want to do a zero-inflated Poisson regression for the data you describe, instead of two separate regressions.

1. Perneger, T.V. What's wrong with Bonferroni adjustments. BMJ 1998; 316 : 1236
2. Cook, R.J. & Farewell, V.T. Multiplicity considerations in the design and analysis of clinical trials. Journal of the Royal Statistical Society, Series A 1996; Vol. 159, No. 1 : 93-110
3. Rothman, K.J. No adjustments are needed for multiple comparisons. Epidemiology 1990; Vol. 1, No. 1 : 43-46
4. Marshall, J.R. Data dredging and noteworthiness. Epidemiology 1990; Vol. 1, No. 1 : 5-7
5. Greenland, S. & Robins, J.M. Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Epidemiology 1991; Vol. 2, No. 4 : 244-251
• Cont'd: 2. Cook R.J. and farewell V.T. Multiplicity considerations in the design and analysis of clinical trials. Journal of the Royal Statistical Society, Series A 1996; Vol. 159, No. 1 : 93-110 – Brenden Oct 4 '10 at 20:41
• Thank you for your comments, Brenden, especially the last one about prediction vs. causal explanation. And welcome to the site! I hope to see many more of your contributions in the future. – whuber Oct 4 '10 at 20:44
• Cont'd: 3. Rothman K.J. No adjustments are needed for multiple comparisons. Epidemiology 1990; Vol. 1, No. 1 : 43-46 4. Marshall J.R. Data dredging and noteworthiness. Epidemiology 1990; Vol. 1, No. 1 : 5-7 5. Greenland S. and Robins J.M. Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Epidemiology 1991; Vol. 2, No. 4 : 244-251 – Brenden Oct 4 '10 at 20:45
• (+1) You may be interested in the following thread: stats.stackexchange.com/questions/3252/…. It seems we share a lot of links in common :-) – chl Oct 4 '10 at 21:14

There are good answers here. Let me add a couple of small points that I don't see covered elsewhere.

First, what is the nature of your response variables? More specifically, are they understood as related to each other? You should only do two separate multiple regressions if they are understood to be independent (theoretically) / if the residuals from the two models are independent (empirically). Otherwise, you should consider a multivariate regression. ('Multivariate' means >1 response variable; 'multiple' means >1 predictor variable.)

The other thing to bear in mind is that the model comes with a global $F$ test, which is a simultaneous test of all the predictors. It is possible that the global test is 'non-significant' while some of the individual predictors appear to be 'significant'. That should give you pause, if it occurs. On the other hand, if the global test suggests at least some of the predictors are related, that gives you some protection from the problem of multiple comparisons (i.e., it suggests not all nulls are true).

You can do a seemingly unrelated regression and use an F test. Put your data in a form like this:

Out1 1 P11 P12 0  0   0
Out2 0 0   0   1  P21 P22


so that the predictors for your first outcome have their values when that outcome is the y variable and 0 otherwise and vice-versa. So your y is a list of both outcomes. P11 and P12 are the two predictors for the first outcome and P21 and P22 are the two predictors for the second outcome. If sex, say, is a predictor for both outcomes, its use to predict outcome 1 should be in a separate variable/column when predicting outcome 2. This lets your regression have different slopes/impacts for sex for each outcome.

In this framework, you can use standard F testing procedures.