5
$\begingroup$

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter adjusted.

Indeed, for an infinite sample, ROC curve is essentially the plot of probability of correctly classifying positive outcomes, $P_{11}(\vec{\theta})$ vs. the probability of correctly classifying negative outcomes, $P_{00}(\vec{\theta})$. In the simplest case, for every observation we simply have probabilities $P_{11}^{(i)},P_{00}^{(i)}$, and we can obtain the ROC curve by counting the number of observations where both probabilities exceed certain threshold: $$ TP = \sum_i I\left[P_{11}^{(i)}>\theta_0\right], TN= \sum_i I\left[P_{00}^{(i)}>\theta_0\right]. $$ E.g., this is how ROC curves are constructed for any classifier in scikit-learn.

However, one can imagine a fixed threshold $\theta_0$, while the probabilities depend on other parameter(s) of the classifier - with a different result.

Perhaps, my understanding of ROC curves is too "intuitive". I will appreciate a discussion from the point of view of fundamental principles and/or references to solid texts.

$\endgroup$
1
  • 2
    $\begingroup$ Yes, the ROC curve depends on the model (and therefore on any parameters the model contains). Given a model, you vary a classification threshold to obtain the curve. $\endgroup$ Nov 30, 2023 at 10:03

1 Answer 1

8
$\begingroup$

Maybe I don't understand your notation, but I feel you are mixing up true probabilities, model outputs (which might be probability estimates) and events.

given an event E, model output $f(\theta)$ (with parameters $\theta$), we choose a threshold $t$ and define detection as $f(\theta)\ge t$. $$P_{11}^{(i)}=P[(f^i(\theta) >=t) \cap E]$$

$$P_{00}^{(i)}=P[(f^i(\theta) <t) \cap \not E]$$

note that the threshold is on the model output, not on the true probabilities - the model output doesn't have to be a probability.

So having cleared that up, different model parameters will naturally give different model outputs and therefore different true positive rates for a given threshold $t$. When one uses an ROC curve, we are fixing a given model with given ([trained] parameters).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.