Is adjusting p-values in a multiple regression for multiple comparisons a good idea?

Question

Lets assume you are a social science researcher/econometrician trying to find relevant predictors of demand for a service. You have 2 outcome/dependent variables describing the demand (using the service yes/no, and the number of occasions). You have 10 predictor/independent variables that could theoretically explain the demand (e.g., age, sex, income, price, race, etc). Running two separate multiple regressions will yield 20 coefficients estimations and their p-values. With enough independent variables in your regressions you would sooner or later find at least one variable with a statistically significant correlation between the dependent and independent variables.

My question: is it a good idea to correct the p-values for multiple tests if I want to include all independent variables in the regression? Any references to prior work are much appreciated.

Answer 1

It seems your question more generally addresses the problem of identifying good predictors. In this case, you should consider using some kind of penalized regression (methods dealing with variable or feature selection are relevant too), with e.g. L1, L2 (or a combination thereof, the so-called elasticnet) penalties (look for related questions on this site, or the R penalized and elasticnet package, among others).

Now, about correcting p-values for your regression coefficients (or equivalently your partial correlation coefficients) to protect against over-optimism (e.g. with Bonferroni or, better, step-down methods), it seems this would only be relevant if you are considering one model and seek those predictors that contribute a significant part of explained variance, that is if you don't perform model selection (with stepwise selection, or hierarchical testing). This article may be a good start: Bonferroni Adjustments in Tests for Regression Coefficients. Be aware that such correction won't protect you against multicollinearity issue, which affects the reported p-values.

Given your data, I would recommend using some kind of iterative model selection techniques. In R for instance, the stepAIC function allows to perform stepwise model selection by exact AIC. You can also estimate the relative importance of your predictors based on their contribution to $R^2$ using boostrap (see the relaimpo package). I think that reporting effect size measure or % of explained variance are more informative than p-value, especially in a confirmatory model.

It should be noted that stepwise approaches have also their drawbacks (e.g., Wald tests are not adapted to conditional hypothesis as induced by the stepwise procedure), or as indicated by Frank Harrell on R mailing, "stepwise variable selection based on AIC has all the problems of stepwise variable selection based on P-values. AIC is just a restatement of the P-Value" (but AIC remains useful if the set of predictors is already defined); a related question -- Is a variable significant in a linear regression model? -- raised interesting comments (@Rob, among others) about the use of AIC for variable selection. I append a couple of references at the end (including papers kindly provided by @Stephan); there is also a lot of other references on P.Mean.

Frank Harrell authored a book on Regression Modeling Strategy which includes a lot of discussion and advices around this problem (§4.3, pp. 56-60). He also developed efficient R routines to deal with generalized linear models (See the Design or rms packages). So, I think you definitely have to take a look at it (his handouts are available on his homepage).

References

1. Whittingham, MJ, Stephens, P, Bradbury, RB, and Freckleton, RP (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75, 1182-1189.
2. Austin, PC (2008). Bootstrap model selection had similar performance for selecting authentic and noise variables compared to backward variable elimination: a simulation study. Journal of Clinical Epidemiology, 61(10), 1009-1017.
3. Austin, PC and Tu, JV (2004). Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. Journal of Clinical Epidemiology, 57, 1138–1146.
4. Greenland, S (1994). Hierarchical regression for epidemiologic analyses of multiple exposures. Environmental Health Perspectives, 102(Suppl 8), 33–39.
5. Greenland, S (2008). Multiple comparisons and association selection in general epidemiology. International Journal of Epidemiology, 37(3), 430-434.
6. Beyene, J, Atenafu, EG, Hamid, JS, To, T, and Sung L (2009). Determining relative importance of variables in developing and validating predictive models. BMC Medical Research Methodology, 9, 64.
7. Bursac, Z, Gauss, CH, Williams, DK, and Hosmer, DW (2008). Purposeful selection of variables in logistic regression. Source Code for Biology and Medicine, 3, 17.
8. Brombin, C, Finos, L, and Salmaso, L (2007). Adjusting stepwise p-values in generalized linear models. International Conference on Multiple Comparison Procedures. -- see step.adj() in the R someMTP package.
9. Wiegand, RE (2010). Performance of using multiple stepwise algorithms for variable selection. Statistics in Medicine, 29(15), 1647–1659.
10. Moons KG, Donders AR, Steyerberg EW, and Harrell FE (2004). Penalized Maximum Likelihood Estimation to predict binary outcomes. Journal of Clinical Epidemiology, 57(12), 1262–1270.
11. Tibshirani, R (1996). Regression shrinkage and selection via the lasso. Journal of The Royal Statistical Society B, 58(1), 267–288.
12. Efron, B, Hastie, T, Johnstone, I, and Tibshirani, R (2004). Least Angle Regression. Annals of Statistics, 32(2), 407-499.
13. Flom, PL and Cassell, DL (2007). Stopping Stepwise: Why stepwise and similar selection methods are bad, and what you should use. NESUG 2007 Proceedings.
14. Shtatland, E.S., Cain, E., and Barton, M.B. (2001). The perils of stepwise logistic regression and how to escape them using information criteria and the Output Delivery System. SUGI 26 Proceedings (pp. 222–226).

Answer 2

To a great degree you can do whatever you like provided you hold out enough data at random to test whatever model you come up with based on the retained data. A 50% split can be a good idea. Yes, you lose some ability to detect relationships, but what you gain is enormous; namely, the ability to replicate your work before it is published. No matter how sophisticated the statistical techniques you bring to bear, you will be shocked at how many "significant" predictors wind up being entirely useless when applied to the confirmation data.

Bear in mind, too, that "relevant" for prediction means more than a low p-value. That, after all, only means it's likely a relationship found in this particular dataset is not due to chance. For prediction it's actually more important to find the variables that exert substantial influence on the predictand (without over-fitting the model); that is, to find the variables that are likely to be "real" and, when varied throughout a reasonable range of values (not just the values that might occur in your sample!), cause the predictand to vary appreciably. When you have hold-out data to confirm a model, you can be more comfortable provisionally retaining marginally "significant" variables that might not have low p-values.

For these reasons (and building on chl's fine answer), although I have found stepwise models, AIC comparisons, and Bonferroni corrections quite useful (especially with hundreds or thousands of possible predictors in play), these should not be the sole determinants of which variables enter your model. Do not lose sight of the guidance afforded by theory, either: variables having strong theoretical justification to be in a model usually should be kept in, even when they are not significant, provided they do not create ill-conditioned equations (e.g., collinearity).

NB: After you have settled on a model and confirmed its usefulness with the hold-out data, it's fine to recombine the retained data with the hold-out data for final estimation. Thus, nothing is lost in terms of the precision with which you can estimate model coefficients.

Answer 3

I think this is a very good question; it gets to the heart of the contentious multiple testing "problem" that plagues fields ranging from epidemiology to econometrics. After all, how can we know if the significance we find is spurious or not? How true is our multivariable model?

In terms of technical approaches to offset the likelihood of publishing noise variables, I would heartily agree with 'whuber' that using part of your sample as training data and the rest as test data is a good idea. This is an approach that gets discussed in the technical literature, so if you take the time you can probably find out some good guidelines for when and how to use it.

But to strike more directly at the philosophy of multiple testing, I suggest you read the articles I reference below, some of which support the position that adjustment for multiple testing is often harmful (costs power), unnecessary, and may even be a logical fallacy. I for one do not automatically accept the claim that our ability to investigate one potential predictor is inexorably reduced by the investigation of another. The family-wise Type 1 error rate may increase as we include more predictors in a given model, but so long as we do not go beyond the limits of our sample size, the probability of Type 1 error for each individual predictor is constant; and controlling for family-wise error does not illuminate which specific variable is noise and which is not. Of course, there are cogent counter-arguments as well.

So, as long as you limit your list of potential variables to those which are plausible (ie, would have known pathways to the outcome) then the risk of spuriousness is already handled fairly well.

However, I would add that a predictive model is not as concerned with the "truth-value" of its predictors as a causal model; there may be a great deal of confounding in the model, but so long as we explain a large degree of the variance then we aren't too concerned. This makes the job easier, at least in one sense.

Cheers,

Brenden, Biostatistical Consultant

PS: you may want to do a zero-inflated Poisson regression for the data you describe, instead of two separate regressions.

1. Perneger, T.V. What's wrong with Bonferroni adjustments. BMJ 1998; 316 : 1236
2. Cook, R.J. & Farewell, V.T. Multiplicity considerations in the design and analysis of clinical trials. Journal of the Royal Statistical Society, Series A 1996; Vol. 159, No. 1 : 93-110
3. Rothman, K.J. No adjustments are needed for multiple comparisons. Epidemiology 1990; Vol. 1, No. 1 : 43-46
4. Marshall, J.R. Data dredging and noteworthiness. Epidemiology 1990; Vol. 1, No. 1 : 5-7
5. Greenland, S. & Robins, J.M. Empirical-Bayes adjustments for multiple comparisons are sometimes useful. Epidemiology 1991; Vol. 2, No. 4 : 244-251

Answer 4

There are good answers here. Let me add a couple of small points that I don't see covered elsewhere.

First, what is the nature of your response variables? More specifically, are they understood as related to each other? You should only do two separate multiple regressions if they are understood to be independent (theoretically) / if the residuals from the two models are independent (empirically). Otherwise, you should consider a multivariate regression. ('Multivariate' means >1 response variable; 'multiple' means >1 predictor variable.)

The other thing to bear in mind is that the model comes with a global $F$ test, which is a simultaneous test of all the predictors. It is possible that the global test is 'non-significant' while some of the individual predictors appear to be 'significant'. That should give you pause, if it occurs. On the other hand, if the global test suggests at least some of the predictors are related, that gives you some protection from the problem of multiple comparisons (i.e., it suggests not all nulls are true).

Answer 5

You can do a seemingly unrelated regression and use an F test. Put your data in a form like this:

Out1 1 P11 P12 0  0   0
Out2 0 0   0   1  P21 P22

so that the predictors for your first outcome have their values when that outcome is the y variable and 0 otherwise and vice-versa. So your y is a list of both outcomes. P11 and P12 are the two predictors for the first outcome and P21 and P22 are the two predictors for the second outcome. If sex, say, is a predictor for both outcomes, its use to predict outcome 1 should be in a separate variable/column when predicting outcome 2. This lets your regression have different slopes/impacts for sex for each outcome.

In this framework, you can use standard F testing procedures.