When to use Simple Linear Regression over Multiple Linear Regression

Question

I am fairly new to the world of statistics and approaching it as I learn more about machine learning. I have a fairly firm grasp on regression analysis so far but not necessarily on nuances and best practices of application.

For example; assume I have 5 predictor variables—a clear case for consideration of multiple regression as I understand it.

I'm curious as to any conditions in which it would be beneficial to draw primary conclusions based on simple linear regression modeling from these data vs. using multiple regression.

The one situation I can imagine is where all five of the explanatory variables are realized to have a high degree of collinearity and can be combined into a single feature.

The only other case I've been able to imagine is where, after initial analyzes for correlation between predictors and the response variable, it's concluded that only a single predictor has any significant correlation such that a linear relationship exists between it and the response variable. In this case, however, it's really a conclusion that only one predictor variable is suited for inclusion anyway—kind of sidestepping the issue.

So the question is: under what conditions would one choose simple linear regression over multiple linear regression when multiple predictor variables are available for analysis. As a caveat, assume more than 1 exists where there exists a significant linear correlation between that variable and the response variable.

Answer 1

If you care about prediction, then you want the model that will maximize your out of sample predictive accuracy. The best way is to have a sense, in advance, of what variables will do that (e.g., all, some, or just one of your variables), and then fit that model. Often, people don't. In such a case, you can use a cross validation scheme to select the model that will perform best (again, that model may use all, some, or just one of your variables. You can read some of our existing threads that discuss cross validation by searching on the cross-validation tag; you may want to sort by votes and start reading from the best ranked threads.

If you have specific hypotheses to test, you should fit the model that corresponds to what you want to know. That model may use all, some, or only one of your variables. Once the model is fit, you can assess it (e.g., use plots to determine if the assumptions are met). If you are OK with the model, you can interpret the hypothesis tests of interest. If it turns out that some covariates are not significant, that's no problem. You do not 'have to', nor should you, drop non-significant variables and refit the model.

Regarding multicollinearity, if you have perfect multicollinearity, you need to choose some variables to drop, or your software will choose for you. Perfect multicollinearity is rare, though, and if you don't have perfect multicollinearity, you don't have to drop any variables. If you care about prediction, multicollinearity may not pose much of a problem. If you care about testing a hypothesis, and the hypothesis of interest isn't part of the collinear variables, it won't matter. If it is part of the collinear variables, it's likely you won't have enough information to answer your question. Dropping variables may lead to a significant p-value, but that p-value won't be valid—it's still the case that you wouldn't have enough information to answer your question, it's just that p-hacking is an effective way to get significant p-values.