I have a relatively short monthly time series (7 years). I'm wondering if I estimate an OLS model with 6 years of data and do pseudo-out of sample forecasting with the remaining year, would the RMSE will be of any use or is thats just not enough data?
The RMSE will only be an unbiased estimate of the true forecasting error the very first time the out of sample data is used to evaluate the prediction.
In practice, I have found that it is very easy to find a model that by chance alone makes good predictions given you test enough models.
As a thought experiment consider if you had a class of models that just made predictions at random. As you test a larger and larger number of these models, you would find that some of them make great out-of-sample predictions. However, the same models would immediately deteriorate when predicting real data.