0
$\begingroup$

We can use ROC or PR curve to access the performance of the classifier,especially on imbalance data. But it is a curve with parameter threshold, even if we get a high ROC or PR performance, which point (or threshold) on there we should choose in order to get a class label for test set FINALLY? And more clearly, even if we get another not high ROC or PR performance for another classifier, maybe there exists a point (threshold) on it with the similar true-positive/false-positive ( precision/recall for positive class) with the previous good one, which means if we choose this threshold, these two classifiers has similar performance on the test set( even if there overall performance are different). Correct? Or we don't choose threshold, we just choose p=0.5 for labeling for a probability classifier?

$\endgroup$
1
$\begingroup$

For a choosen value (T) of the threshold you can compute the False alarm rate (FAR) and the Hit rate (HR). Obviously, both quantities depend on the threshold T, i.e. FAR(T), HR(T).

One finds the ROC curve by drawing the couples ( FAR(T), HR(T) ) for vaying values of T in a planar graph.

Consequently, as the ROC curve is derived from the set of ''all values of T'' it is a global measure of performance.

One can only conclude on the better performance of one or other test if one of the two ROCs ''dominates'' the other one. If the ROC curves intersect there is no unambigous conclusion.

So, as you say, the ROC is a global measure of performance.

$\endgroup$
  • $\begingroup$ I would add that the ROC is independent of the threshold and so is useful for comparing the general performance of classifiers. After you fix the thresholds, then one classifier may be better than another, even if it has a worse ROC curve. I.e. it may be better locally while be worse globally. $\endgroup$ – dcorney Jul 28 '15 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.