Can predictive power be inferred from only in-sample modelling results?

I wonder if one can tell anything about predictive power of a model if model selection and estimation was done using all available data. That is, there was no data left for "out of sample" prediction trials. (In other words, data was not split into a "training sample" to select and estimate the model and a "test/holdout sample" to examine forecasting performance.)

In other words, can predictive power be inferred from only in-sample modelling results?

There are several subtle points I think you missed:

1. In a mechanistic world there may be a true model (e.g., Newtonian mechanics) but it other situations this is very unlikely. Instead one usually finds that when more and better data become available one can get a better model. BIC assumes there is a single true model.
2. Selecting a model according to the best BIC is a biased process that will result in overfitting.
3. $R_{adj}^2$ was designed to de-bias an ordinary $R^2$ in a linear model when there is a single pre-specified model.

I believe that the only thing you can assume is that if your sample is very large and is representative of the future to which you want to apply a forecast, the apparent $R^2$ is an upper bound of the real $R^2$.

• Great answer! Let me see if I understand you arguments. (1): I appreciate the possibility that the "true" model may have evolved over time. Is this what you mean by more & better data -> better model? If so, we should denote this time-dependent model as the "true" model. (Surely, there must have been a process that generated the data - we call it the "true" process - regardless of what we know about it. It might have been a very complex one, of course, so it might be missing in the pool of our candidate models.) If so, there is nothing wrong with BIC assuming a single true model. Sep 24 '14 at 14:20
• (2): Model selection by BIC is biased because the choice between all k-parameter models in the pool of candidate models is due to explanatory power. That is, BIC does data mining (data snooping?) inside the set of k-parameter models, for each k. Other than that, BIC is consistent. Did I understand it correctly? Sep 24 '14 at 14:23
• (3) I would only use R^2_adj for the ultimately selected model. If I was certain that the model selection procedure was sound, it would make sense to use R^2_adj, wouldn't it? Again, the R^2_adj would not be used in the model selection stage, but only in the last stage to assess the properties of the ultimate model. Sep 24 '14 at 14:26
• Don't write a bunch of separate comments. $R_{adj}^{2}$ is not for an "ultimately selected model". It is for a single pre-specified model. BIC, if there were to be one true model, can help you select the most probable model. The probability that this model is the true model may still be 0.01 and the resulting selected model can be terribly overfitted. Sep 24 '14 at 14:59
• In my experience BIC does not meet its goal. This is most pronounced when solving for an optimal shrinkage parameters - BIC will over-shrink making the calibration on future data poor from too much under-fitting. I am still troubled about the use of BIC to select from a large number of competing models. I think this will at least result in over-interpretation, and it will not compete with a penalized "super model" that is a superset of all the models entertained. Sep 25 '14 at 20:29