I have two OLS models and want to do an out of sample prediction for wages on a test set. In the first model I excluded the insignificant variable. The second model has the insignificant variable. The second model yields a better prediction because it has a smaller mse then the first model. How could that be? What's the explanation for this result? Have you maybe a reference where somebody had the same results for prediction with significant and insignificant coefficients?
If you are interested in prediction accuracy then in general I'd advise you to not be that concerned with the statistical significance of predictors.
There may be certain predictors that you think should definitely be significant (for example, if you were modeling ice cream sales and had a variable for time of year), and if they are not, that may indicate some kind of specification issue.
I also like to include rules if I expect certain variables to have a specific sign (eg, temperature should have a positive effect on ice cream sales)-- if the sign is wrong there is some kind of issue.
And finally I also like to use rules for the magnitude of certain coefficients. For example, if predicting the weekly sales of televisions, a variable representing thanksgiving/black friday week should not only be positive but it should be at least 2 because the sales will at least be doubled.
Concern over the significance of predictors is more appropriate when you are concerned with the effect of x on y, rather than the prediction of y. Some may disagree with me here-- I think there's some philosophical differences in the approach in an academic/theoretical setting and in industry where you simply must make a prediction and want to do whatever gives the best possible prediction against a holdout/validation set.
We should note that p-values a measure of strength against the hypothesis that a given coefficient is zero. Note that this does not directly give us any statement about the predictive power of the given coefficient. For example, in truth it may be a very powerful predictor but we have not collected evidence in regard to this coefficient, so really there's no strong reason to think that a model with insignificant predictors should do worse than one with only significant predictors.
In general, p-values should not be seen as tool to improve predictive models. Building accurate predictive models leads to very different strategies, such as tuning complex penalized models with cross-validation.
The significance of a predictor corresponds to an "added last" test; given all the variables in the model, how much variability is explained in the outcome by adding the predictor in question? If the other predictors in the model are highly associated with the predictor in question, then there is little it will add to the prediction of the outcome, even if on its own it is a good predictor of the outcome.
For a set of highly correlated predictors, it may be that the added last tests for each of them are nonsignificant because each individually doesn't do more than the variables already in the model, even though together they explain the outcome well. So, it's possible for a set of predictors to jointly explained the outcome well even if none or few of the predictors are significant.
because of this phenomenon, you should not consider the significance of a variable in determining whether to include it in a predictive model. You might use other variable selection techniques that are separate from statistical significance if you want a parsimonious model. Otherwise, you might end up discarding a well-performing model just because many of its predictors are nonsignificant, which could be an artifact of the fact that significance depends not only on preditive ability but also on correlation with other predictors in the model.