Which data split should I use to determine cutoff point for classification? I'm building a classification model using the caret package. I'm splitting my dataset in train and test (80/20) and training using 10-fold cross-validation repeated 10 times to find the best parameters. 
After the model is trained, I want to pick the best cutoff point (based on my criteria). Should I do it using the train set, the test set, or should I split on a third set just to optimize the cutoff?
From what I understand, the test set should only be used to get a better performance estimate. I shouldn't use it to optimize for anything, be it the cutoff or even choosing the best model. If that's correct, it would mean that I need another dataset to determine the cutoff, am I right?
 A: EDIT: @topepo, author of the book cited here, pointed me to http://topepo.github.io/caret/custom_models.html#Illustration5 where he answered this exact question. Basically, he adds the cutoff points as another tuning parameter and uses the standard caret functions to select the best option.
Max Kuhn writes on his Applied Predictive Modeling book, 1st edition, page 425:

(...) It is important, especially for small
  samples sizes, to use an independent data set to derive the cutoff. If
  the training set predictions are used, there is likely a large
  optimistic bias in the class probabilities that will lead to
  inaccurate assessments of the sensitivity and specificity. If the test
  set is used, it is no longer an unbiased source to judge model
  performance. For example, Ewald (2006) found via simulation that post
  hoc derivation of cutoffs can exaggerate test set performance.

The highlight is mine. Based on this, it seems that I have to split my dataset in train/evaluate/test sets, and define the cutoff based on the evaluate set.
A: It somehow depends. If you assess the performance of the model with a measure that requires determining cut point before (like accuracy), then you should not use test set to determine cut point. However, if you would use a measure that does not require determining cut point to assess model performance - let's say AUC - you absolutely can choose model basing on final test set AUC and then determine optimal cut point using your test set. And that is solution I would recommend as optimal.
A: Never train on your test set. You only use the test set to report your classification error. You can cross validate the training set to tune the parameters and select the best model. After that you report the performance on the test set and that should only be done at the end. 
If you tune your parameters w.r.t. the test set, then the performance of the model is not truly reflected on unseen data.
