I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score diverges from my understanding:

roc_auc_score([1, 0, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]) # output: 0.5

As far as I undestand, ROC curve can be defined in the following way: $$ROC(x) = \max \{ TPR(\mathcal{C}_t) \mid FPR(\mathcal{C}_t) \leq x \}$$

Where $TPR$ and $FPR$ is true-positive rate and false-positive rate and $\mathcal{C}_t$ is a classifier with fixed threshold value equals to $t$.

In case where all predicted probabilities are equal we have only two different classifiers: one that always predict one class, and one that always predict another class. In this case we have two corner in our ROC-curve: $(0, 0)$ and $(1, 1)$. And by definition above the curve has a shape of corner:

corner shape

But why sklearn draws curve that shapes single line like this?

single line shape

How should I change the definition of curve to get this kind of picture?


It looks like we simply need to extend our family of classifiers. We are able to do this with a little bit of randomization. So, we can define classifier $\mathcal{C}_t^p$ in the following way: \begin{equation} \mathcal{C}_t^p(x) = \begin{cases} \texttt{+1}&, \texttt{if } C(x) > t\\ \texttt{-1}&, \texttt{if } C(x) < t\\ \texttt{+1 with probability } p \texttt{ and -1 with } 1-p&, \texttt{if } C(x) = t\\ \end{cases} \end{equation}

After this we can simply adjust our definition of ROC-curve:

$$ROC(x) = \max \{ TPR(\mathcal{C}_t^p) \mid p \in [0..1] \wedge FPR(\mathcal{C}_t^p) \leq x \}$$

It perfectly make sense with only single correction that current TPR, FPR are expected values but not deterministic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.