Say you build a model performing <grid/random/hyperband/bayesian/anyothermethod> search hiperparameter tuning with K-FOLD crossvalidation, where K>=3. Then you get the best estimator (the one that has the best </loss/accuracy/f1/auc/anyothermetric> ), then you train with all data and get that there is a 50% of difference between train or test (overfitted model). You still go with that model, test it on another dataset( backtest) and you get good results like in test. Then does overfitting matters? I mean would you accept this overfitted model for production or you woud prefer a more regularized model? If you do so, why? Would you prefer an underfitted model that has no significant difference between train and test but worse than an overfitted model that has better results on test ? Of course I would love to see some papers/books that talk about the topic.


I saw this question (validation/training accuracy and overfitting ) and the chart is very useful, another way to put this would be, looking at the chart how much complexity would you choose for a model in production.

Thanks in advance.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.