1
$\begingroup$

I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The second model yields a better prediction because it has a smaller mse then the first model. How could that be? What's the explanation for this result? Have you maybe a reference where somebody had the same results for prediction with significant and insignificant coefficients?

$\endgroup$
4
$\begingroup$

If you are interested in prediction accuracy then in general I'd advise you to not be that concerned with the statistical significance of predictors.

There may be certain predictors that you think should definitely be significant (for example, if you were modeling ice cream sales and had a variable for time of year), and if they are not, that may indicate some kind of specification issue.

I also like to include rules if I expect certain variables to have a specific sign (eg, temperature should have a positive effect on ice cream sales)-- if the sign is wrong there is some kind of issue.

And finally I also like to use rules for the magnitude of certain coefficients. For example, if predicting the weekly sales of televisions, a variable representing thanksgiving/black friday week should not only be positive but it should be at least 2 because the sales will at least be doubled.

Concern over the significance of predictors is more appropriate when you are concerned with the effect of x on y, rather than the prediction of y. Some may disagree with me here-- I think there's some philosophical differences in the approach in an academic/theoretical setting and in industry where you simply must make a prediction and want to do whatever gives the best possible prediction against a holdout/validation set.

$\endgroup$
  • $\begingroup$ And how about a sample selection bias problem in a wage estimation. I mean if i do two ols regressions one with sample selection correction by using lambda from the heckit method (inverse Mills'Ratio) and one without the sample selection correction term. If lambda is not signifcant it means i have no sample selection bias. If I do with both models an out of sample prediction and the model with the insignifcant lambda yield a better mse value how can explain the result? $\endgroup$ – MasterStudent1992 May 19 at 21:15
  • $\begingroup$ I’m not familiar with that. You said insignificant lambda = no bias so wouldn’t you expect better prediction? Nevertheless, to apply the same principles I mentioned above if something in the model results/output doesn’t make sense then pay attention to it. If your sample is biased and prediction is better then perhaps your out of sample set is also biased. Or perhaps it was random chance, and you should not assume such “luck” in the future. $\endgroup$ – Chris Umphlett May 19 at 21:45
  • $\begingroup$ insignficant lambda means the regression coeffcients are not biased so no sample selection correction is needed en.wikipedia.org/wiki/Heckman_correction $\endgroup$ – MasterStudent1992 May 19 at 22:04
2
$\begingroup$

We should note that p-values a measure of strength against the hypothesis that a given coefficient is zero. Note that this does not directly give us any statement about the predictive power of the given coefficient. For example, in truth it may be a very powerful predictor but we have not collected evidence in regard to this coefficient, so really there's no strong reason to think that a model with insignificant predictors should do worse than one with only significant predictors.

In general, p-values should not be seen as tool to improve predictive models. Building accurate predictive models leads to very different strategies, such as tuning complex penalized models with cross-validation.

$\endgroup$
2
$\begingroup$

The significance of a predictor corresponds to an "added last" test; given all the variables in the model, how much variability is explained in the outcome by adding the predictor in question? If the other predictors in the model are highly associated with the predictor in question, then there is little it will add to the prediction of the outcome, even if on its own it is a good predictor of the outcome.

For a set of highly correlated predictors, it may be that the added last tests for each of them are nonsignificant because each individually doesn't do more than the variables already in the model, even though together they explain the outcome well. So, it's possible for a set of predictors to jointly explained the outcome well even if none or few of the predictors are significant.

because of this phenomenon, you should not consider the significance of a variable in determining whether to include it in a predictive model. You might use other variable selection techniques that are separate from statistical significance if you want a parsimonious model. Otherwise, you might end up discarding a well-performing model just because many of its predictors are nonsignificant, which could be an artifact of the fact that significance depends not only on preditive ability but also on correlation with other predictors in the model.

$\endgroup$
  • $\begingroup$ The desire for a parsimonious model in this setting is misplaced. $\endgroup$ – Frank Harrell Jun 13 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.