In order to compare a few models to start my ML project, first I split the dataset into train and test, and then performed nested CV on the training set only and got my fair estimate of true risk on my test set. I did all this with sklearn pipeline in order to avoid data leakage.
I then chose a few of the most promising classifiers and I looked at their learning and validation curves. My goal at this point is to pick the ones that seem to have the right approximation and estimation errors and then start creating some features to further refine them.
Assuming this order of things is correct, should I have plotted learning and validation curves against my training set only (by training set I mean the original training set before it was further split again into cross validation sets when I performed nested CV) or should I plot against the entire dataset i.e. training + test set?
I plotted against the training set only and the curves are looking great, but if I plot against the entire dataset they look pretty bad.