2
$\begingroup$

I'm trying to understand ROC-curve and AUC characteristic for it and found that behaviour of sklearn function roc_auc_score diverges from my understanding:

roc_auc_score([1, 0, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]) # output: 0.5

As far as I undestand, ROC curve can be defined in the following way: $$ROC(x) = \max \{ TPR(\mathcal{C}_t) \mid FPR(\mathcal{C}_t) \leq x \}$$

Where $TPR$ and $FPR$ is true-positive rate and false-positive rate and $\mathcal{C}_t$ is a classifier with fixed threshold value equals to $t$.

In case where all predicted probabilities are equal we have only two different classifiers: one that always predict one class, and one that always predict another class. In this case we have two corner in our ROC-curve: $(0, 0)$ and $(1, 1)$. And by definition above the curve has a shape of corner:

corner shape

But why sklearn draws curve that shapes single line like this?

single line shape

How should I change the definition of curve to get this kind of picture?

$\endgroup$
2
  • $\begingroup$ Sklearn draws this separating line cause ROC AUC is at least or higher than 0.5. 0.5 value means that your model is totally unreliable and incorrect (at any threshold you cannot correctly divide classes). $\endgroup$ Nov 13, 2021 at 17:25
  • $\begingroup$ @GregoryStelmashenko It's not true what you're saying. You can draw AUROC curves that go below 0.5 I did it. Can't share bc it's a sensitive info. BTW it's isn't necessarily bad to have a AUROC lower than 0.5. AUROC can be used to measure a correlation between a numeric explanatory variable to a binary response. If the explanatory variable is a loan's sum and the response is default (inability to pay), you'd like a AUROC less than 0.5, for example. $\endgroup$
    – Alex Teush
    Dec 7, 2023 at 6:16

1 Answer 1

0
$\begingroup$

It looks like we simply need to extend our family of classifiers. We are able to do this with a little bit of randomization. So, we can define classifier $\mathcal{C}_t^p$ in the following way: \begin{equation} \mathcal{C}_t^p(x) = \begin{cases} \texttt{+1}&, \texttt{if } C(x) > t\\ \texttt{-1}&, \texttt{if } C(x) < t\\ \texttt{+1 with probability } p \texttt{ and -1 with } 1-p&, \texttt{if } C(x) = t\\ \end{cases} \end{equation}

After this we can simply adjust our definition of ROC-curve:

$$ROC(x) = \max \{ TPR(\mathcal{C}_t^p) \mid p \in [0..1] \wedge FPR(\mathcal{C}_t^p) \leq x \}$$

It perfectly make sense with only single correction that current TPR, FPR are expected values but not deterministic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.