1
$\begingroup$

I have about 2,500 records in my dataset and I'm using a 60-20-20 train-validation-test split. I want to generate a lift chart to gauge the efficacy of my model at ranking cases. To date, I've only been plotting lift curves for training and testing (never-before-seen, the latter) datasets. Should I include in the training line on the chart my validation set, as well, since the model technically "learned from" (i.e. learned the best model hyperparameters from) that dataset as well?

$\endgroup$
  • $\begingroup$ Nope. Just use the testing set. Also given the data set size mentioned is not huge probably using k-fold CV instead of a fixed train-validation split is more robust. $\endgroup$ – usεr11852 says Reinstate Monic Feb 13 at 0:19
1
$\begingroup$

If you want to include the testing results on a data set whose test results were used for optimization (your "validation" set), I'd recommend to do that in a third, separate lift curve (same for any other figure of merit or diagram).

Yes, optimization is part of the model training and the model did learn from these results. We expect the optimization lift curve to be somewhere in between the test results for training (in the narrower sense) and truly independent test sets.

Where the results on the validation (optimization) set are in relation to training and test set results can help you to find whether the optimization didn't go well and/or whether there's trouble with the training in the narrower sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.