Does too many variables in a regression model affect inference?

Question

Regression models can be used for inference on the coefficients to describe predictor relationships or for prediction about an outcome. I'm aware of the bias-variance tradeoff and know that including too many variables in the regression will cause the model to overfit, making poor predictions on new data. Do these overfitting problems extend to inferences made on the predictors?

Say I'm working with a cancer dataset (n=200) that includes overall survival and several hundred genomic markers. I'm interested in describing the relationship between each marker and survival, and would like to identify markers that show strong evidence of an association with survival. Is it wrong to fit a model with all the markers and clinical factors (age, sex, treatment etc) and then look at hazard ratios, confidence intervals, and p-values to identify "important" predictors? Building a model with hundreds of parameters feels wrong, but I'm not sure if there's an underlying reason why this approach should be avoided. Would this create a multiple comparisons problem? Does sample size play a role in whether this approach is valid?

In my experience some people would use stepwise model selection (using p-values or AIC) to identify important predictors based on the final p-values, but from what I've read stepwise selection overexaggerates p-values and provides unreliable inference due to selection bias. I also try to avoid building univariable models for each predictor because omitted variable bias can create misleading effects estimates.

The results from my model would be hypothesis generating to prioritize gene candidates for experimental study.

Answer 1

One problem with dumping all of your predictors into the model is the invitation to extreme collinearity, which will inflate your standard errors and likely make your results uninterpretable.

Judea Pearl has pointed to a second problem, if your inference is aimed at modeling causal relationships. In trying to "control for everything" by including all available predictors, you may actually "unblock" new confounder paths and move farther away from, not closer to, good estimates of causal relationships. In the language of his graphical system, you create a confound if you condition on a collider or on a descendant of a collider.

A third problem, with your limited sample size, statistical power with so many predictors will be low, which will inflate the likelihood that what seems like a finding now will prove not to be later on, following the reasoning of John Ioannidis (2005).

Answer 2

I am certainly not an expert in cancer research, but have read that genomic markers (also known as genetic markers) have been studied with respect to their relationship to various diseases.

To unmask the true strength of such genetic markers, I suspect, one might want to control for exposure to agents likely associated with cancer especially relating to geographic locations. The latter can serve as proxies for areas of elevated pollution either in the air, water or the food stock, or places noted for fresh fruits (like mangos,...) believed to be beneficial to the immune system. Other control factors could be age (as a nonlinear variable to proxy the general strength of the immune system). Also, income levels, may be a proxy for healthcare access and a possible indicator for those starting point with better health.

Also, my research on chemotherapy and survival rates, revealed a positive benefit that may arise from one's ancestry. For example, ancestors of people who endured long ocean voyages seeking new habitats (a darwinian survival of the fittest) apparently have better cancer survival rates following chemotherapy.

Controlling for such factors may make the statistical process of assessing the impact of genetic markers more accurate (and also may allow MORE such factors to be included in the model). Here are some studies mentioning some of the controlling factors .