# A ''significant variable'' that does not improve out-of-sample predictions - how to interpret?

I have a question that I think will be quite basic to a lot of users.

Im using linear regression models to (i) investigate the relationship of several explanatory variables and my response variable and (ii) predict my response variable using the explanatory variables.

One particular explanatory variable X appears to signficantly impact my response variable. In order to test the added value of this explanatory variable X for the purpose of the out-of-sample predictions of my response variable I used two models: model (a) which used all explanatory variables and model (b) which used all variables except variable X. For both models I solely report the out-of-sample performance. It appears that both models perform almost identically as good. In other words, adding the explanatory variable X does not improve out-of-sample predictions. Note that I also used model (a), i.e. the model with all explanatory variables, to find that explanatory variable X does significantly impact my response variable.

My question now is: how to inpret this finding? The straightforward conclusion is that, even though the variable X appears to significantly influence my response variable using inferential models, it does not improve the out-of-sample predictions. However, I have trouble further explaining this finding. How can this be possible and what are some explanations for this finding?

Extra information: with 'significantly influence' I mean that 0 is not included in the highest 95% posterior density interval of the parameter estimate (im using a Bayesian approach). In frequentist terms this roughly corresponds to having a p-value lower than 0.05. I am using only diffuse (uninformative) priors for all my models parameters. My data has a longitudinal structure and contains around 7000 observations in total. For the out-of-sample predictions I used 90% of the data to fit my models and 10% of the data to evaluate the models using multiple replications. That is, I performed the train-test split multiple times and eventually report the average performance metrics.

• Because you are using a Bayesian approach, your results depend as much on your prior as on the data. Because the dependence on the prior decreases with increasing amounts of data and increases to the extent the data and prior disagree, it would be useful for you to supply information both about the prior distribution, the amount of data, and how closely the data alone conform to the prior distribution.
– whuber
Aug 25, 2018 at 13:23
• @whuber I forgot to mention that I am only using diffuse (uninformative) priors. Therefore, I do not feel like my prior specification has anything to do with my findings. I'm pretty sure that fitting frequentist linear regression models will result in the exact same findings. Aug 25, 2018 at 13:44
• Thanks--that helps rule out several possible explanations.
– whuber
Aug 25, 2018 at 14:17
• Are you refitting the models to the held out data or using the models you fit to your original data? In either case one possible problem is that you are making a Type II error on the held out data; perhaps the variable is relevant but you were underpowered originally (in which case you are probably overestimating the effect which could make predictions worse). Or the variable was irrelevant and you made a Type I error. There are lots of reasons this type of thing might happen.
– guy
Aug 25, 2018 at 16:00
• I have used several metrics: RSME, MAE and AUC (Im also trying to predict whether my depedent variable, which is continous, is below a certain threshold). Aug 29, 2018 at 10:27

When a particular predictor is statistically significant doesn't really mean that it also considerably improves the predictive performance of a model. Predictive performance is more related to the effect size. As an example, the function below simulates data from a linear regression model with two predictors x1 and x2, and fits two models, one with both x1 and x2, and one with x1 alone. In the function you can change the effect size for x2. The function reports the confidence intervals for the coefficients of x1 and x2, and the $R^2$ values of the two models as a measure of predictive performance.

The function is:

sim_ES <- function (effect_size = 1, sd = 2, n = 200) {
# simulate some data
DF <- data.frame(x1 = runif(n, -3, 3), x2 = runif(n, -3, 3))
DF$y <- 2 + 5 * DF$x1 + (effect_size * sd) * DF$x2 + rnorm(n, sd = sd) # fit the models with and without x2 fm1 <- lm(y ~ x1 + x2, data = DF) fm2 <- lm(y ~ x1, data = DF) # results list("95% CIs" = confint(fm1), "R2_X1_X2" = summary(fm1)$r.squared,
"R2_only_X1" = summary(fm2)$r.squared) }  As an exampple, for the default values we get, $95% CIs
2.5 %   97.5 %
(Intercept) 1.769235 2.349051
x1          4.857439 5.196503
x2          1.759917 2.094877

$R2_X1_X2 [1] 0.9512757$R2_only_X1
[1] 0.8238826


So x2 is significant, and not including it in the model has a big impact on the $R^2$.

But if we set the effect size to 0.3, we get:

> sim_ES(effect_size = 0.3)
$95% CIs 2.5 % 97.5 % (Intercept) 1.9888073 2.5563233 x1 4.9383698 5.2547929 x2 0.3512024 0.6717464$R2_X1_X2
[1] 0.9542341

$R2_only_X1 [1] 0.9450327  The coefficient is still significant but the improvement in the$R^2$is very small. • The vague dichotomy between statistical significance vs predictive performance is the bane of my analytics life in more ways than one. (+1 - and a general welcome to CV Prof.!) Sep 13, 2018 at 21:48 This is a fairly normal thing to happen in multiple regression. The most common reason is that your predictors are related to one another. In other words, you can infer X from the values of the other predictors. Therefore, while it's useful for predictions if it's the only predictor you have, once you have all the other predictors it doesn't provide much extra information. You can check whether this is the case by regressing X on the other predictors. I would also refer to the chapter on linear regression in the free online textbook, Elements of Statistical Learning. • You seem to be describing a non-significant explanatory variable rather than addressing the specific circumstances described in the question. – whuber Aug 25, 2018 at 14:18 • I'm describing an explanatory variable which is significantly related to the response on its own (i.e. in a simple regression), which is what I presume the question means by "X appears to signficantly impact my response variable". Aug 25, 2018 at 14:30 • But in that case I would not have found that my explanatory variable X significantly impacts my response variable right? Maybe I did not make it clear in my question initially, but I used a model with all explanatory variables to find that explanatory variable X has a significant influence on my response variable. Aug 25, 2018 at 15:36 • I read the question as meaning that$X\$ is significant in the context of a multiple regression. This seems pretty clear from the references to "several explanatory variables." I am concerned that your answer might be confusing the OP.
– whuber
Aug 25, 2018 at 17:47
• Yes whuber, you understood it correctly. This is what I mean. I hopefully clarified this well enough in my question. Aug 25, 2018 at 19:51