Timeline for How Cross Validation detect overfitting?
Current License: CC BY-SA 4.0
17 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 20, 2023 at 13:40 | comment | added | Dikran Marsupial | BTW I think that the phenomenon you were describing already has a standard terminology in statistics - namely the "bias-variance trade-off". The variance bit refers to a model capturing the variability in the data due to random sampling, i.e. "spurious signals" and the bias part measures how well the model has captured the underlying structure of the data generating process. If you are overfitting it means that the variance is too high for the level of bias. Underfitting means the bias is too high and the model may improve if variance were allowed to increase. | |
| Jul 20, 2023 at 13:02 | comment | added | Dikran Marsupial | @gazza89 sorry that is at best selective quoting, a fuller quote is "the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit to additional data or predict future observations reliably" which specifically relates it to generalisation performance and thus matches the definition I gave, not the semantics you are using. What does "underfitting" mean with your semantics? | |
| Jul 20, 2023 at 12:49 | comment | added | gazza89 | If you really want to get pedantic, the spread of definitions you will find for overfitting when you google it will tend to follow the Wikipedia definition of "fitting the data too closely", i.e. "finding spurious signal" I agree that the more consequential point is the one around finding the point of optimal model fittedness where over- and underfitting are ultimately traded off. Nonetheless, understanding how this comes about under the hood is something I personally find useful. | |
| Jul 20, 2023 at 12:31 | comment | added | Dikran Marsupial | @gazza89 the "picking up false signal" semantics is not useful because it would mean you can't have an underfitted model. The fact that you can't fit a model to noisy data without picking up some noise means that it is meaningless to describe it as "over"fitting as any amount of picking up noise is too much "over". Better to use semantics that are useful rather than confusing and consistent with related terminology. | |
| Jul 20, 2023 at 12:24 | comment | added | gazza89 | @mainakmukherjee: Cross-validation is a process which allows you to compare many models to one and other, and then you can choose the one you think will generalise best to unseen data. If you want a model which better trades off "picking up signal Vs picking up noise", that's more of a question of model design. | |
| Jul 20, 2023 at 12:21 | comment | added | gazza89 | @DikranMarsupial: it really depends on the semantics of whether you mean "picking up false signal" or "fitting beyond the point of optimal complexity" I've tried to make the difference very clear in my response | |
| Jul 20, 2023 at 12:21 | comment | added | mainak mukherjee | Can you please give me an example how cross validation (alone or with other techniques) can detect/prevent overfitting ? | |
| Jul 20, 2023 at 12:03 | comment | added | Dikran Marsupial | Mrs Marsupial and I wrote a paper on that topic that you might find useful jmlr.org/papers/volume11/cawley10a/cawley10a.pdf | |
| Jul 20, 2023 at 12:02 | comment | added | Dikran Marsupial | Cross-validation is a method of estimating the generalisation (operational) performance of a model generated by a particular procedure. It isn't in itself a method of avoiding overfitting. If you tune your model and monitor the cross-validation performance, you can detect the point where the model becomes too complex and it starts to overfit the data, because at that point the cross-validation error will start to get worse rather than better. Note however it is possible to over-fit cross-validation as well | |
| Jul 20, 2023 at 11:54 | comment | added | mainak mukherjee |
The thing understand about k fold cross validation is we split the data in k folds then train the data iteratively k times with k-1 folds and leave 1 for validation. It is still not clear for me how it can say about overfitting ? I am using sklearn cross_val_score to find out the score of cross validation for each iteration. It is returning an array of the scores. What is the point of using cross validation on a data ? Please explain, from morning I am torning my hairs to find out these answers.
|
|
| Jul 20, 2023 at 11:31 | comment | added | Dikran Marsupial | "Cross validation doesn't detect overfitting" this isn't true either. If as you train the model or tune the hyper-parameters the cross-validation error gets worse while the training error gets better, you have detected overfitting. That is why early stopping can be effective. It doesn't always detect it with 100% reliability, but it does detect over-fitting most of the time. Note you can over-fit the cross-validation error as well if you tune too much. | |
| Jul 20, 2023 at 11:27 | comment | added | Dikran Marsupial | "I would say that as soon as you start to fit a model, you are starting to overfit. " is not correct. Overfitting means improving the fit to the training sample in a way that decreases generalisation performance. While you are improving generalisation performance, then by definition you are not overfitting. The fact that you can have an underfit model shows that not all models overfit from the start. How can you have a model that is simultaneously both under- and over-fitted? | |
| Jul 20, 2023 at 11:27 | comment | added | gazza89 | To be honest, I don't fully understand what you mean by a cross validation score of 0.96. There are two numbers you should be reporting, your train loss and your test loss. It sounds like 0.96 is your train loss ? (as an aside, what kind of model are you using?) | |
| Jul 20, 2023 at 10:09 | vote | accept | mainak mukherjee | ||
| Jul 20, 2023 at 10:08 | comment | added | mainak mukherjee | Given my example, the cross validation score is really good, but the model is still overfitted ?. Am I right?? | |
| Jul 20, 2023 at 9:41 | history | edited | Roger V. | CC BY-SA 4.0 |
added 4 characters in body
|
| Jul 20, 2023 at 8:55 | history | answered | gazza89 | CC BY-SA 4.0 |