11
$\begingroup$

Regression models can be used for inference on the coefficients to describe predictor relationships or for prediction about an outcome. I'm aware of the bias-variance tradeoff and know that including too many variables in the regression will cause the model to overfit, making poor predictions on new data. Do these overfitting problems extend to inferences made on the predictors?

Say I'm working with a cancer dataset (n=200) that includes overall survival and several hundred genomic markers. I'm interested in describing the relationship between each marker and survival, and would like to identify markers that show strong evidence of an association with survival. Is it wrong to fit a model with all the markers and clinical factors (age, sex, treatment etc) and then look at hazard ratios, confidence intervals, and p-values to identify "important" predictors? Building a model with hundreds of parameters feels wrong, but I'm not sure if there's an underlying reason why this approach should be avoided. Would this create a multiple comparisons problem? Does sample size play a role in whether this approach is valid?

In my experience some people would use stepwise model selection (using p-values or AIC) to identify important predictors based on the final p-values, but from what I've read stepwise selection overexaggerates p-values and provides unreliable inference due to selection bias. I also try to avoid building univariable models for each predictor because omitted variable bias can create misleading effects estimates.

The results from my model would be hypothesis generating to prioritize gene candidates for experimental study.

$\endgroup$
1
  • $\begingroup$ Yes, sample size will play a role, because you will reduce your cell size with each new variable you add. Additionally, with this many variables, you are likely to introduce some collinearity along the way, which can cloud your conclusions. $\endgroup$ Jan 7, 2020 at 21:11

2 Answers 2

7
$\begingroup$

One problem with dumping all of your predictors into the model is the invitation to extreme collinearity, which will inflate your standard errors and likely make your results uninterpretable.

Judea Pearl has pointed to a second problem, if your inference is aimed at modeling causal relationships. In trying to "control for everything" by including all available predictors, you may actually "unblock" new confounder paths and move farther away from, not closer to, good estimates of causal relationships. In the language of his graphical system, you create a confound if you condition on a collider or on a descendant of a collider.

A third problem, with your limited sample size, statistical power with so many predictors will be low, which will inflate the likelihood that what seems like a finding now will prove not to be later on, following the reasoning of John Ioannidis (2005).

$\endgroup$
2
  • 1
    $\begingroup$ Your 3rd point alludes an inflated error rate with a limited sample size. Is the error rate dependent on statistical power? I thought the two were independent so as long as you're not p-hacking (repeated testing, stepwise selection etc) the error rate should be maintained. Does the error rate change as statistical power changes? $\endgroup$ Jan 8, 2020 at 1:26
  • 2
    $\begingroup$ Ioannidis (2005) argued that, when you get a "significant" effect when power is low, this is more likely because of an extreme sample / false positive, which will not replicate in independent samples. If further research is based on different data, I believe teh same logic would apply here. $\endgroup$
    – Ed Rigdon
    Jan 8, 2020 at 12:33
0
$\begingroup$

I am certainly not an expert in cancer research, but have read that genomic markers (also known as genetic markers) have been studied with respect to their relationship to various diseases.

To unmask the true strength of such genetic markers, I suspect, one might want to control for exposure to agents likely associated with cancer especially relating to geographic locations. The latter can serve as proxies for areas of elevated pollution either in the air, water or the food stock, or places noted for fresh fruits (like mangos,...) believed to be beneficial to the immune system. Other control factors could be age (as a nonlinear variable to proxy the general strength of the immune system). Also, income levels, may be a proxy for healthcare access and a possible indicator for those starting point with better health.

Also, my research on chemotherapy and survival rates, revealed a positive benefit that may arise from one's ancestry. For example, ancestors of people who endured long ocean voyages seeking new habitats (a darwinian survival of the fittest) apparently have better cancer survival rates following chemotherapy.

Controlling for such factors may make the statistical process of assessing the impact of genetic markers more accurate (and also may allow MORE such factors to be included in the model). Here are some studies mentioning some of the controlling factors .

$\endgroup$
1
  • $\begingroup$ I agree that it's important to adjust for strong prognostic factors such as environmental exposures to get better estimates for the genomic markers. Too many studies seem to only include the genomic markers in their model and forget to include strong prognostic factors like treatment or tumor stage. Other answers here seem to suggest that too many variables does bias inference, so I think the question becomes is the exposure or ancestry important enough to include in the model and worth the extra bias. $\endgroup$ Jan 8, 2020 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.