Best model in-sample is the worst out-of-sample

Question

I have three times series (X, Y and Z) with a length of 2000 observations. I am fitting three different models:

• $Y_t = \beta_0 + \beta_1 Y_{t-1} + \epsilon_t$;
• $Y_t = \beta_0 + \beta_1 Y_{t-1} + \beta_2 X_{t-1} + \epsilon_t$;
• $Y_t = \beta_0 + \beta_1Y_{t-1} + \beta_2 Z_{t-1} + \epsilon_t$.

The adjusted $R^2$ of the second model is identical to that of the first model ($0.607)$, while the adjusted $R^2$ from the third is higher ($0.638$). However, when I employ a rolling window of a thousand observations and compute the one-step-ahead forecasts for the three models, comparing them by $MSE$, there is no significant difference in performance. This is counterintuitive to me, since the in-sample $R^2$ from the third model is significantly higher than that of the second, while no regressor is being added (hence there is no over-fitting).

Answer 1

no regressor is being added (hence there is no over-fitting)

This is where you are going wrong. Model B can be overfit compared to model A even if the models are non-nested. The argument that "overfitting can only occur when predictors are added" is incorrect.

As an illustration, imagine that you are randomly generating a time series $(Z_t)$ and fitting your third model... and that you are doing this many times, and keeping the one realization of $(Z_t)$ that gave you the highest $R^2$ in-sample. Since the $(Z_t)$ is random, its future realizations will not improve your forecasts, and the high $R^2$ is simply due to overfitting compared to any other model.