# What terms should I include in a linear regression model?

Consider the following OLS output. My question is: If you have to define a linear equation from the regression, do you then only include significant variables? If so, the equation according to the SPSS output (DV = stockVol0LN) would be:

stockVol0LN = 0.632 * StockVol1LN + 0.278 * StockVol2LN + 0.152 * hbVol0LN -
0.050 * hbVol1LN - 0.069 * hbVol2LN - 0.045 * hbAgreeQuality0LN +
0.064 * wiki0LN - 0.067 * wiki1LN + 0.050 * svi0 + 1.396


My question is: Is this the best equation, or is there another measure besides significance level to consider when deciding what terms you should include in the model?

-
This is a question about selection. It is a common topic here & elsewhere. You should not only include significant variables, to understand why, you may want to read this: algorithms-for-automatic-model-selection. You should also search / read around CV under the tags: feature-selection, model-selection, & stepwise-regression, for starters. If there is still something that you want to know after having read through that, edit your Q here to clarify, otherwise this Q should be closed as a FAQ. –  gung Aug 27 '12 at 20:02
"Best" equation for what purpose? Prediction, estimation, understanding, theory-building? –  whuber Aug 27 '12 at 21:27
@whuber prediction :-) –  Pr0no Aug 27 '12 at 22:11
OK, then in your researches pay special attention to mentions of holding out data to use for verifying the model as well as more sophisticated versions known as "cross validation." Using these approaches will make you somewhat immune from the dangers of many model selection procedures such as stepwise regression. –  whuber Aug 27 '12 at 22:15

Modeling questions should be based first on the science that underlies the question with computer output used as a helpful tool.

The above information in not enough to come to a final model, and may not be enough for the next step.

It is possible for 2 variables to both be non-significant when adjusted for other terms in the model, but be very important when the other is deleted. Consider if you have as predictors height in inches and height in centimeters (with some rounding so that they are not exactly the same). Models that include both predictors will probably tell you that both are redundant given the other, but remove one and the other may be very important (removing both because of high p-values would be a major mistake).

You can also have variables that work synergistically, each by themselves is not very predictive, but together they are. The diameter of the arteries in the neck (which may be related to anurisms and other blood-brain conditions) is related to the difference between systolic blood pressure and diastolic blood pressure. Either measure on its own may have only a weak relationship, but together it can be much stronger.

Also consider this case, you have 2 predictors X1 and X2, the statistical analysis suggests that you only need one of the 2 in your model and that X1 does a slightly better job of predicting your Y variable (maybe an $R^2=0.81$ vs. $R^2=0.80$), but X2 is something quick, easy, and non-invasive to measure (temperature, or blood pressure) while X1 is the result of a lab test that takes several hours on a biopsy that requires major surgery to obtain (and would not be done most times other than to collect X1); which is the better predictor to use?

Before deciding on a final model you need to spend more time deciding why you are doing the modeling (understanding relationships, prediction, etc.) and the science behind the question and data. You should probably spend more time in a regression class or with a textbook (just ignoring the "nonsignificant" predictors is not the same as fitting the model with only the significant predictors).

-
@MichaelChernick, while I join you in not being as negative about stepwise regression as Peter and Frank Harrell are, I think Peter is probably referring to the fact that the stepwise selection you've described messes up the usual interpretation of the $p$-values that are left after selection. But, depending on why you're doing the stepwise selection, this may not matter e.g. if your goal is prediction and you're deleting terms based on drop in out-of-sample prediction accuracy. –  Macro Aug 27 '12 at 21:18