Timeline for Is the best model always one with best test score, even though it looks overfit?
Current License: CC BY-SA 4.0
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 30, 2018 at 12:21 | comment | added | Ugur MULUK | Just a little addition, we may want slightly overfit models, since our performance may depend on catching some of the outlier data as well, while not too much concentrating on them. Still depending on the task, you may want to catch only extreme points as the task which have strong deviation from the distribution, then you need completely overfit model. This field of science has some ill-defined parts, involving abstractness. | |
| Nov 30, 2018 at 12:17 | comment | added | Ugur MULUK | I get your point, but that depends on your task or metrics ,or even definition of performing better . For example, I have a LightGBM model same as you where overfit model performs better (less accuracy and precision as metric) but does better what I want to do. Rather than getting the fit point, you may train at small N iterations and retrain the same model. between N iterations you can check your model, if it is doing what you need, or where it is going. Also, you can find a better metric. For some tasks, classic interpretations can be useless, or deceiving. | |
| Nov 30, 2018 at 12:10 | comment | added | Auren Ferguson | I use early stopping. The point is on the test set (separate to train and validation) the overfit performs better. I'm not sure which is the better model, the one that is best on unseen data or the one where train and validation are closer to each other. | |
| Nov 30, 2018 at 12:01 | history | answered | Ugur MULUK | CC BY-SA 4.0 |