How do we know that a model (really) has a predictive power?

Let's assume that we have a trained predictive model and some data set to validate / evaluate the model. We also have a measure of accuracy (mean squared deviation). We apply the given model to a given data set and get a value of our measure (let's say 1.37). Now, how can we determine if the model has any predictive power?

Intuitively, I would say that a model has a predictive power, if it can use (at list a little bit) the input features to generate predictions that tend to be closer to targets. In that context I would check if the given model performs better than the best model that does not use features per construction (it means that the reference model always "predicts" the same number).

Now, the next question is what number should I take as a "prediction" of the reference model? Should it be in-sample mean (mean of the target for the data set that was used for the training of the to be evaluated model) or out-of-sample mean (mean of the target for the data set that is used for the evaluation)?

Next, let's assume that the evaluated model gives better (smaller) mean squared deviation that the "constant" model. Does it really mean that the evaluated model has a predictive power? Could it be that it is better just by chance (and it will not be better for another evaluation data set)? How can I calculate statistical significance of the error to be smaller than the reference error?

• It is common to compare with respect to a baseline model, and the baseline can be a constant mean model. In fact this is the idea behind $R^2$. The correct way to measure prediction accuracy is to observe out of sample prediction errors. The probability of "luckiness/unluckiness" goes lower as we calculate out of sample accuracy on more and more samples. – Cagdas Ozgenc Dec 7 '18 at 7:56
• @CowboyTrader, my problem is that I have a "border case" since I do not have enough samples. Therefore I would do a statistical significance test to determine if the error is really smaller and it is not just a fluctuation. Moreover, it is still not clear to me what mean should I use (in- or -out-of-sample) as a reference "model". – Roman Dec 7 '18 at 10:04
• Normally you should use the out-of-sample performance for both the candidate model and the baseline model. When the number of data points for out-of-sample is scarce, it is better to use information criteria such as AIC, AICc. This is an adjustment for in-sample bias. You simply use all available samples for training (no out-of-sample). After that you may simply compare AIC for candidate model and baseline model and choose the one with smaller AIC. – Cagdas Ozgenc Dec 7 '18 at 12:08