Is ROC curve unique?

Question

ROC curve and the area under it (AUC) are routinely used to evaluate the performance of binary classifiers. However, it seems that both, the shape of the curve and the area, depend on the parameter adjusted.

Indeed, for an infinite sample, ROC curve is essentially the plot of probability of correctly classifying positive outcomes, $P_{11}(\vec{\theta})$ vs. the probability of correctly classifying negative outcomes, $P_{00}(\vec{\theta})$. In the simplest case, for every observation we simply have probabilities $P_{11}^{(i)},P_{00}^{(i)}$, and we can obtain the ROC curve by counting the number of observations where both probabilities exceed certain threshold: $$ TP = \sum_i I\left[P_{11}^{(i)}>\theta_0\right], TN= \sum_i I\left[P_{00}^{(i)}>\theta_0\right]. $$ E.g., this is how ROC curves are constructed for any classifier in scikit-learn.

However, one can imagine a fixed threshold $\theta_0$, while the probabilities depend on other parameter(s) of the classifier - with a different result.

Perhaps, my understanding of ROC curves is too "intuitive". I will appreciate a discussion from the point of view of fundamental principles and/or references to solid texts.

Answer 1

Maybe I don't understand your notation, but I feel you are mixing up true probabilities, model outputs (which might be probability estimates) and events.

given an event E, model output $f(\theta)$ (with parameters $\theta$), we choose a threshold $t$ and define detection as $f(\theta)\ge t$. $$P_{11}^{(i)}=P[(f^i(\theta) >=t) \cap E]$$

$$P_{00}^{(i)}=P[(f^i(\theta) <t) \cap \not E]$$

note that the threshold is on the model output, not on the true probabilities - the model output doesn't have to be a probability.

So having cleared that up, different model parameters will naturally give different model outputs and therefore different true positive rates for a given threshold $t$. When one uses an ROC curve, we are fixing a given model with given ([trained] parameters).