In order to compare a few models to start my ML project, first I split the dataset into train and test, and then performed nested CV on the training set only and got my fair estimate of true risk on my test set. I did all this with sklearn pipeline in order to avoid data leakage.

I then chose a few of the most promising classifiers and I looked at their learning and validation curves. My goal at this point is to pick the ones that seem to have the right approximation and estimation errors and then start creating some features to further refine them.

Assuming this order of things is correct, should I have plotted learning and validation curves against my training set only (by training set I mean the original training set before it was further split again into cross validation sets when I performed nested CV) or should I plot against the entire dataset i.e. training + test set?

I plotted against the training set only and the curves are looking great, but if I plot against the entire dataset they look pretty bad.


Learning Curves are meant for training test sets CV only and they give you the feel of the best hypreparameters to build your model that will be tested in real life for many cases in your entire data. If you want Learning Curves to be awesome and fine with your entire data , that means you tuned all your data to that model and now your Learning Curves shows that you overfit your model to entire data. You should not use Learning Curves on the entire dataset.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.