1
$\begingroup$

I have a relatively short monthly time series (7 years). I'm wondering if I estimate an OLS model with 6 years of data and do pseudo-out of sample forecasting with the remaining year, would the RMSE will be of any use or is thats just not enough data?

$\endgroup$
3
0
$\begingroup$

The RMSE will only be an unbiased estimate of the true forecasting error the very first time the out of sample data is used to evaluate the prediction.

In practice, I have found that it is very easy to find a model that by chance alone makes good predictions given you test enough models.

As a thought experiment consider if you had a class of models that just made predictions at random. As you test a larger and larger number of these models, you would find that some of them make great out-of-sample predictions. However, the same models would immediately deteriorate when predicting real data.

$\endgroup$
3
  • $\begingroup$ What do you mean by the very first time? As in if I'm choosing variables in order to lower the RMSE it will no longer be unbiased? $\endgroup$
    – rinnovo
    Jun 14 '15 at 21:18
  • $\begingroup$ Correct. If you build one model with one set of parameters, then the first time you use the out of sample data to validate your model will provide an unbiased estimate of RMSE. Changing the variables included in your model or changing your hyperparameters and retesting leads to a biased RMSE. Now in very large datasets, this bias can be very small but even then it still exists and is why many people intentionally hold out one last validation dataset to test at the very end. $\endgroup$ Jun 14 '15 at 23:20
  • $\begingroup$ Ok thats helpful. How is one supposed to compare different variables predictive power than? Accept that the RMSE is biased? Also, how many observations do you think are needed to make the RMSE worth looking at, even the first time? Is 12 too little? $\endgroup$
    – rinnovo
    Jun 15 '15 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.