# Statistical approaches to detect overfitting in simple models

I read here that there are statistical approaches to assess whether a tractable machine learning model (e.g., a linear regression model) overfits a dataset:

Simpler models that have originated in statistics often don't need test datasets. Instead, what degree the model is overfit can be calculated directly as statistical significance: a ‘p-value’. These statistical methods are powerful, well established, and form the foundation of modern science. The advantage is that the training set doesn't ever need to be split and we get a much more precise understanding of how confident we can be about a model.

That is a huge advantage: we don't need to split the training data or perform cross-validation if the model is simple enough.

Unfortunately I haven't seen such a thing in practice. I know we get a p-value for each fitted coefficient when fitting regression models like here, but I don't believe these p-values determine overfitting.

Do you have a more elaborate explanation and preferably some examples of this use case?

• A p-value is not a measure of overfitting! We have a lot of good threads that explain what it is and its uses & abuses. Here's a good place to start: stats.stackexchange.com/questions/31/…
– mkt
Commented Jul 2 at 14:16
• @mkt so what do you make of what the webpage says?
– Rafs
Commented Jul 19 at 10:36
• I think it fundamentally misunderstands p-values and is borderline incoherent. It is far from alone in misunderstanding p-values though. But the basic point that there are ways of quantifying how well a simple model fits other than those use in machine learning is reasonable.
– mkt
Commented Jul 19 at 11:48

## 1 Answer

The various information criteria (Bayesian, Aikake) and e.g. Cp try to estimate this. So for example, the Bayesian information criterion (https://en.m.wikipedia.org/wiki/Bayesian_information_criterion) measures fit by likelihood, except the final score is also penalised if you have more parameters because then you're more likely to be overfitting. Mallow's Cp and the Aikake Information Criterion works on similar principles but are tweaked slightly differently.

However, if you have access to a computer, cross validate. Cross validation is the solution to the problem.

• Thank you. It'd be appreciated if you can expand your first sentence more.
– Rafs
Commented Jul 8, 2022 at 16:06
• Sorry I haven't seen that you added more clarification to your answer till now! So with simple models, model selection can tell if a model is overfitting. However, it seems that this is applicable to complex models too; besides, I don't see how p-values are utilised here - reflecting on the source in my question..
– Rafs
Commented Jun 30, 2023 at 9:54