2
$\begingroup$

I read here that there are statistical approaches to assess whether a tractable machine learning model (e.g., a linear regression model) overfits a dataset:

Simpler models that have originated in statistics often don't need test datasets. Instead, what degree the model is overfit can be calculated directly as statistical significance: a ‘p-value’. These statistical methods are powerful, well established, and form the foundation of modern science. The advantage is that the training set doesn't ever need to be split and we get a much more precise understanding of how confident we can be about a model.

That is a huge advantage: we don't need to split the training data or perform cross-validation if the model is simple enough.

Unfortunately I haven't seen such a thing in practice. I know we get a p-value for each fitted coefficient when fitting regression models like here, but I don't believe these p-values determine overfitting.

Do you have a more elaborate explanation and preferably some examples of this use case?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

The various information criteria (Bayesian, Aikake) and e.g. Cp try to estimate this. So for example, the Bayesian information criterion (https://en.m.wikipedia.org/wiki/Bayesian_information_criterion) measures fit by likelihood, except the final score is also penalised if you have more parameters because then you're more likely to be overfitting. Mallow's Cp and the Aikake Information Criterion works on similar principles but are tweaked slightly differently.

However, if you have access to a computer, cross validate. Cross validation is the solution to the problem.

$\endgroup$
2
  • $\begingroup$ Thank you. It'd be appreciated if you can expand your first sentence more. $\endgroup$
    – Rafs
    Jul 8, 2022 at 16:06
  • $\begingroup$ Sorry I haven't seen that you added more clarification to your answer till now! So with simple models, model selection can tell if a model is overfitting. However, it seems that this is applicable to complex models too; besides, I don't see how p-values are utilised here - reflecting on the source in my question.. $\endgroup$
    – Rafs
    Jun 30, 2023 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.