I have run a multiple linear regression using stepwise regression to select the best model, however the best model returned has a non-significant variable. When I remove this the AIC value goes up indicating the model without the significant variable is a worse fit. Should I remove the non-significant predictor or should I leave it in as it is a better model?
$\begingroup$
$\endgroup$
4
-
5$\begingroup$ What is your goal here? Prediction or explanation? How big is your data? $\endgroup$– TrynnaDoStatCommented Jan 18, 2015 at 15:09
-
1$\begingroup$ Thanks for your answer below. My goal is prediction and my data set has 590 cases. $\endgroup$– PoppyCommented Jan 18, 2015 at 16:44
-
2$\begingroup$ I give a detailed list of the problems with stepwise model building in my answer here: stats.stackexchange.com/questions/115843/…. $\endgroup$– AlexisCommented Jan 19, 2015 at 1:01
-
1$\begingroup$ Suppressor variables are often not significant, yet they can affect fit-statistics a lot. Might be interesting to check for suppressor effects (especially with a view to model interpretation). $\endgroup$– StatisticsRatCommented Mar 15, 2017 at 16:35
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Leave it in. The data are incapable of really telling you which model is "better" unless you use AIC in a highly structured way (e.g. on a pre-specified large group of variables), and removing insignificant variables invalidates the estimate of $\sigma^2$ and all $P$-values, standard errors, and confidence limits in addition to invalidating the formula for adjusted $R^2$. Much is written about these issues on this site.
answered Jan 18, 2015 at 15:28
$\begingroup$
$\endgroup$
NB: A corollary to Frank Harrell's answer is that stepwise variable selection should not be used in the first place. That is, not only is it a mistake to discard that final 'leftover' non-significant covariate, but it was even more wrong to employ an automated procedure (stepwise variable selection) designed to produce a cascade of many such mistakes very quickly in an interdependent and irreproducible fashion.
$\begingroup$
$\endgroup$
You need to test your model on multiple test datasets. AIC is measure of model fitting not accuracy. Please read section 3.3 ( Subset Selection) in this book -
http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf
removing variables is suggested because of 2 reasons-
The first is prediction accuracy: keeping all variables often have low bias but large variance. Prediction accuracy can sometimes be improved by shrinking or setting some coefficients to zero. By doing so we sacrifice a little bit of bias to reduce the variance of the predicted values, and hence may improve the overall prediction accuracy.
The second reason is interpretation. With a large number of predictors, we often would like to determine a smaller subset that exhibit the strongest effects. In order to get the “big picture,” we are willing to sacrifice some of the small details.
answered Mar 15, 2017 at 16:28