2
$\begingroup$

I'm building a classification model using the caret package. I'm splitting my dataset in train and test (80/20) and training using 10-fold cross-validation repeated 10 times to find the best parameters.

After the model is trained, I want to pick the best cutoff point (based on my criteria). Should I do it using the train set, the test set, or should I split on a third set just to optimize the cutoff?

From what I understand, the test set should only be used to get a better performance estimate. I shouldn't use it to optimize for anything, be it the cutoff or even choosing the best model. If that's correct, it would mean that I need another dataset to determine the cutoff, am I right?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

EDIT: @topepo, author of the book cited here, pointed me to http://topepo.github.io/caret/custom_models.html#Illustration5 where he answered this exact question. Basically, he adds the cutoff points as another tuning parameter and uses the standard caret functions to select the best option.

Max Kuhn writes on his Applied Predictive Modeling book, 1st edition, page 425:

(...) It is important, especially for small samples sizes, to use an independent data set to derive the cutoff. If the training set predictions are used, there is likely a large optimistic bias in the class probabilities that will lead to inaccurate assessments of the sensitivity and specificity. If the test set is used, it is no longer an unbiased source to judge model performance. For example, Ewald (2006) found via simulation that post hoc derivation of cutoffs can exaggerate test set performance.

The highlight is mine. Based on this, it seems that I have to split my dataset in train/evaluate/test sets, and define the cutoff based on the evaluate set.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You can also use resampling to set it as if it is a tuning parameter. Here is an example. $\endgroup$
    – topepo
    Commented Jul 29, 2015 at 20:12
1
$\begingroup$

It somehow depends. If you assess the performance of the model with a measure that requires determining cut point before (like accuracy), then you should not use test set to determine cut point. However, if you would use a measure that does not require determining cut point to assess model performance - let's say AUC - you absolutely can choose model basing on final test set AUC and then determine optimal cut point using your test set. And that is solution I would recommend as optimal.

Improve this answer
$\endgroup$
0
$\begingroup$

Never train on your test set. You only use the test set to report your classification error. You can cross validate the training set to tune the parameters and select the best model. After that you report the performance on the test set and that should only be done at the end.

If you tune your parameters w.r.t. the test set, then the performance of the model is not truly reflected on unseen data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So I need to split my data into another set and use it to define the cutoff value? $\endgroup$
    – Vítor Baptista
    Commented Jul 29, 2015 at 14:26
  • $\begingroup$ You can do that. I would suggest training on 80% of the data, i.e. do corss-validation, then select the cut-off on the training data and report the error on the test set. $\endgroup$
    – Gumeo
    Commented Jul 29, 2015 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.