So I am looking into a regression model that is supposed to predict the value of a house based on numerous independent variables. What I don't quite understand is how to select the "best" model when eliminating insignificant variables. The original model contained four independant variables of which three turned out to be insignificant (p-value > 0.05). So I removed, say X1, which had the largest p-value (small t-stat). The new model containing three variables clearly appeared to be better, as Adjusted R2 rose (I understand Adjusted R Square is suitable for comparing models with different number of parameters). The F-tests also seemed to confirm that removing X1 was the right thing to do due to its insignificance. However, in the three variable model there was still one insignificant variable, X3. So I removed it as well. Now, in the two variable model, both variables were reliable (small p-values), but Adjusted R Square was actually somewhat lower than that of the previous model. F-tests seemed to indicate that X3 had a small enough impact on the model to be eliminated. Which is actually the better model statistically and economically? The three variable model with the highest Adjusted R Square or the two variable model with no insignificant variables but lower Adjusted R2? I couldn't quite figure out the logic behind this.
-
3$\begingroup$ Why do you want to remove variables in your model? $\endgroup$– Michael MCommented Oct 30, 2013 at 18:04
-
1$\begingroup$ Well I was assigned a task to find the best acceptable regression model using different combinations of predictors. $\endgroup$– GiancarloCommented Oct 31, 2013 at 7:33
-
$\begingroup$ You should check alternative methods to stepwise regression (as it has some problems) like LASSO regression, Ridge regression (selection will not be as evident as with LASSO), SPLS, etc... $\endgroup$– RiffCommented Feb 8, 2017 at 8:35
3 Answers
If your goal is prediction rather than trying to infer causality, there is no need to remove any predictor variables, as @Michael Mayer has already commented, unless there is a significant cost to obtaining the values of some of them. And although your predictor variables may be the "independent" variables in your analysis, it's likely that many are correlated to each other (e.g., number of bedrooms, number of bathrooms, size of lot). In those cases trying to remove variables that are "insignificant" in analysis of a particular sample may lead to weird, counter-intuitive results that do not generalize well for future predictions.
If you do need to remove predictor variables for some reason, follow specific defined methods for hierarchical/stepwise analyses as suggested by @jmbejara, rather than trying to make up the rules as you go. These methods use better ways to compare models than the adjusted R-squared values, and they are available in R and other statistical analysis software.
From what I can tell, it generally depends on your purpose. If you are trying infer causality, then you need choose variables that specify some structure that actually has some theory behind it. Do you actually have good, intuitive reasons for including one variable or the other? (However, this is useless without some sort of identification strategy to disentangle other effects and biases).
On the other hand, if your goal is only to create a model with predictive power, it seems like in general you want to choose variables that make significant contributions to the fit of the model. For prediction and descriptive analysis, it seems to me like different fields have different rules. See this discussion about hierarchical and stepwise regressions of an example of different kinds of rules. In the case you have describe above, there isn't a hard rule to determine which case to choose.
-
$\begingroup$ To make it more confusing, what should I make of a regression coefficient having the "wrong" sign? It's definately significant as far as p-value goes, but the sign is the opposite of what it should be (interestingly, it's a negative coefficient despite there being positive correlation with the dependant variable). There certainly isn't any multicollinearity to be worried about in the model, that's been checked. Do I remove the variable or what? $\endgroup$ Commented Nov 1, 2013 at 18:19
-
$\begingroup$ First, you should know that the correlation between the dependent and independent variable will likely mean nothing once you've included other independent variables. For example, there is a negative correlation between health and going to the hospital, but this doesn't mean that going to the hospital makes people sick. Once you've controlled for the right things, you should see that the coefficient on going to the hospital to be positive. Also, make sure that you're interpreting the regression correctly. It is the effect once all the other variables are held constant. $\endgroup$– jmbejaraCommented Nov 2, 2013 at 4:46
-
$\begingroup$ And just to reiterate what @MichealMayer said, if your goal is to predict, you probably don't care about the signs of the coefficients. You just want to fit the data the best you can. Therefore, you probably shouldn't take anything out. But, again, there's really no hard rules for these things. If you're goal was to think about causality, things would be different. $\endgroup$– jmbejaraCommented Nov 2, 2013 at 4:48
I would have handled your exercise a bit differently. Instead of including all variables and removing one after the other, I would have instead included only the best variable first (the one with the highest absolute correlation with the dependent variable). Next, I would have looked at which of the remaining variables have the highest correlation with the residual of the model with the best variable. That is how to select your second best variable. And, you continue this process as long as your Adjusted R Square keeps on rising.
To go back to your question, which ones of the two models would I choose... I would go with the one with the highest adjusted R Square. That's because a variable does not have to be statistically significant to contribute incremental information to the model. That's as long as the direction or the sign of the coefficient makes sense. Here I disagree with other responders. I think if your regression coefficient is of opposite sign than the original correlation (contradicting economic theory or the logic underlying the model) you are heading for trouble. You will likely have an "overfit" model that will perform very poorly in forecasting.
Also, Goodness-of-fit is not the only criteria to select a model. And, it is an incomplete one at that. You also have to check the performance of your models in a Hold Out sample. You also should conduct sensitivity analysis. I mean by that stress the values of the independent variables associated with possible economic scenarios somewhat outside the norm but that are still possible. And, the best model is often the one that performs best in the Hold Out and through the sensitivity analysis testing. Meanwhile, this same model may have the weaker Goodness-of-fit (lower R Square). This happens all the time. A model with a lower R Square and fewer variables very often beats out one with more variables and a higher R Square... because the latter was simply overfit.
To go back to your original question, you can't select between your two models yet until you test both of them through a Hold Out sample and conduct some sensitivity analysis. Once you do that, a clear winner will readily emerge.
