I'm new to Machine Learning and Statistics so pardon me if I say anything ridiculous.

By "test set" I mean the set that we evaluate the final hypothesis and then report the final result (e.g. test error) that is an unbiased estimate of the corresponding out-of-sample result (e.g. out-of-sample error).

By "validation set" I mean the set that we use to do model selection or parameter tuning to choose out the final hypothesis. The best result found on the validation set is biased by definition (if you evaluate only one hypothesis on the validation set, then the validation set is a test set).

I am sorry for the two above lengthy paragraphs as I want to be sure that we are talking about the same thing. Now come the main question:

Why do we want to calculate ROC curve on test set?

In many other resources that I read, they calculated ROC curve on either training set or test set without a clear definition of "test set", so pardon me if I read it wrong. However, I'm still curious if in the case of the test set by my above definition, what is so the point of calculating ROC curve? Isn't the threshold choice made on the training set (which is perhaps heavily optimistically biased) or the validation set (which might be less optimistically biased)? Doesn't the test set become a validation set if we make threshold choice on it?

The procedure that sounds reasonable to me is that we calculate ROC curve on the validation set to do model selection / parameter tuning and threshold selection based on the ROC curve.

  • 2
    $\begingroup$ The Area Under the Curve (AUC) is a measure of discriminatory power. If you want an out of sample estimate of it then you have to estimate it on the test set? $\endgroup$ – user83346 Mar 6 '17 at 5:32
  • 1
    $\begingroup$ I understand the confusion and think it is mainly caused by semantics. In the field I'm in we usually say 'development'-set to indicate the data which is used for model selection and tuning, while 'validation'-set is the set in which we check the out-of-sample performance or external validity. Regarding your Q, IMO the ROC or AUC as @fcop states, is a reasonable performance measure, but only describes discrimination (can the model discern between two groups), while the AUC does not assess whether the predicted probability are correct (calibration). So yes, you can use it, but not on its own. $\endgroup$ – IWS Mar 6 '17 at 8:15
  • 2
    $\begingroup$ This is not what one normally mean by validation set, which is probably why you're confused. See the following Q&A for the "standard" definitions: stats.stackexchange.com/questions/19048/… $\endgroup$ – Calimo Mar 6 '17 at 8:34
  • $\begingroup$ Thank you very much for your replies, may I ask the "calibration" that @IWS mentioned is the "calibration plot" in this? $\endgroup$ – ntvy95 Mar 7 '17 at 0:47
  • 1
    $\begingroup$ The calibration plot in your reference is indeed the visualization of the "calibration" I referred to. After reading the post @calimo referenced to, I would like to point your and other readers' attention to it as well, as it probably captures CrossValidated's semantics better than my comment above. $\endgroup$ – IWS Mar 7 '17 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.