Let's assume that we have a trained predictive model and some data set to validate / evaluate the model. We also have a measure of accuracy (mean squared deviation). We apply the given model to a given data set and get a value of our measure (let's say 1.37). Now, how can we determine if the model has any predictive power?
Intuitively, I would say that a model has a predictive power, if it can use (at list a little bit) the input features to generate predictions that tend to be closer to targets. In that context I would check if the given model performs better than the best model that does not use features per construction (it means that the reference model always "predicts" the same number).
Now, the next question is what number should I take as a "prediction" of the reference model? Should it be in-sample mean (mean of the target for the data set that was used for the training of the to be evaluated model) or out-of-sample mean (mean of the target for the data set that is used for the evaluation)?
Next, let's assume that the evaluated model gives better (smaller) mean squared deviation that the "constant" model. Does it really mean that the evaluated model has a predictive power? Could it be that it is better just by chance (and it will not be better for another evaluation data set)? How can I calculate statistical significance of the error to be smaller than the reference error?