Statistical approaches to detect overfitting in simple models

Question

I read here that there are statistical approaches to assess whether a tractable machine learning model (e.g., a linear regression model) overfits a dataset:

Simpler models that have originated in statistics often don't need test datasets. Instead, what degree the model is overfit can be calculated directly as statistical significance: a ‘p-value’. These statistical methods are powerful, well established, and form the foundation of modern science. The advantage is that the training set doesn't ever need to be split and we get a much more precise understanding of how confident we can be about a model.

That is a huge advantage: we don't need to split the training data or perform cross-validation if the model is simple enough.

Unfortunately I haven't seen such a thing in practice. I know we get a p-value for each fitted coefficient when fitting regression models like here, but I don't believe these p-values determine overfitting.

Do you have a more elaborate explanation and preferably some examples of this use case?

Answer 1

The various information criteria (Bayesian, Aikake) and e.g. Cp try to estimate this. So for example, the Bayesian information criterion (https://en.m.wikipedia.org/wiki/Bayesian_information_criterion) measures fit by likelihood, except the final score is also penalised if you have more parameters because then you're more likely to be overfitting. Mallow's Cp and the Aikake Information Criterion works on similar principles but are tweaked slightly differently.

However, if you have access to a computer, cross validate. Cross validation is the solution to the problem.