I am working with a data set of ~1200 rows and 60 variables, and I'm trying to build a multiple linear regression model. I do this by separating 10% of the dataset to be used for validation and I use the rest of the data to train the linear model.

I have found that I am able to build very strong regression models, with an adjusted coefficient of determination (R-square) around .998 and <3% of the predicted values trained on the validation set deviating from the true values by more than accepted threshold (also 3%). I also have all the variables selected (~20 variables) as significant at the 5% level, including first-order interaction variables.

However, I have found that by randomly selecting a different 10% of the data as validation, the regression model I trained previously does not consistently have its explanatory variables remain significant, and the proportion of predicted values within the accepted threshold varies, sometimes improving but usually decreasing. The coefficients of the variables also vary. However, the R-square remains extremely high.

How can I be sure to build a stable model? Or rather, what could I reasonably consider a stable model? I'm confused by the apparent disparity between how the R-square and the actual predicted values explain the strength of the model. I'm also concerned that variables previously considered to be significant at the 5% level often fail to remain so. Can anyone give any tips on how to approach model building so that the proportion of the data saved for validation doesn't drastically alter explanatory variables selected?

  • $\begingroup$ Upon further inspection, it appears that the coefficients vary within their standard errors most of the time, so the values of the coefficients themselves remain stable - I'm still unsure about why the significance of the variables might change so much, however. $\endgroup$ – Gabriel Oct 1 '14 at 21:07
  • 1
    $\begingroup$ Remember that under random sampling a relationship with p=.05 has a 50-50 chance of producing a p>.05 among the next sample chosen. Sound like you've overfitted. $\endgroup$ – rolando2 Oct 1 '14 at 23:13

How did you select the variables? If you used a stepwise procedure, you have introduced a bias in the estimation, and the resultant t statistics will give you the wrong results. If you included all first-order interactions for 20 variables, you have estimated another 190 parameters. You don't have enough data to reliably estimate that.

For a stable model, use a shrinkage method like ridge regression or lasso to estimate your model. Alternately, you can do variable selection by doing some flavor of subset regression. Assess model stability using cross-validation.

  • $\begingroup$ (+1) The last sentence is key to reducing variability of out-of-sample performance estimates. $\endgroup$ – Scortchi - Reinstate Monica Oct 2 '14 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.