# Does the output of a binary classification model equal to the probability of a positive outcome?

Assuming you have a binary classification model $$M$$ i.e. that for an input $$x$$ it outputs a number $$M(x)=\hat{y}$$ where $$\hat{y}\in[0,1]$$ predicting the binary label of $$y\in\{0,1\}$$.

For example, a model that receives an image $$x$$ and outputs whether $$x$$ has a cat in it or not.

If such a model $$M$$ has high AUC-ROC (0.9+) for a large test dataset of $$X$$ and $$Y$$, does that mean that $$\hat{y}$$ is, in some way, the probability for $$y=1$$, (or $$P(y=1)=\hat{y}$$)?

Are there any resources (articles, books, etc.) regarding the relationaship between $$P(y=1)$$ and $$\hat{y}$$?

This question touchs the basics of classification models, and yet I couldn't find any good resources about the subject.

### UPDATE:

What helped me better understand the classification vs probability subject is probability calibration which is measured by a metric called expected calibration error.

• The boundary here is between probabilistic modeling and decision theory. It's common for ML folks to decide that y=1 when P(y=1) > threshold, though on this site we seem to agree that thresholding is bad for probabilistic modeling. Commented Nov 1, 2021 at 18:30
• This gets at model calibration, but even if the AUC is low, it is reasonable to say that $\hat y$ is your best estimate of $P(y = 1\vert\text{data})$.
– Dave
Commented Nov 1, 2021 at 18:32
– Sycorax
Commented Nov 1, 2021 at 18:55
• I am not quite sure what are your assumptions, but you could have same AUC with monotonic increasing transformation of your probability estimate ( eg sqrt) Commented Nov 1, 2021 at 21:54
• @Sycorax, thanks, I didn't see the  trick anywhere in the tips above the question text box... it is very useful! Commented Nov 2, 2021 at 6:56

In theory, this is true. Start with logistic regression. That explicitly models the log-odds, which you can convert to the probability. A neural network with a sigmoid activation function on the final node is behaving the same as the inverse link function in a logistic regression. You're trying to get the probability.
Many machine learning models have poor calibration. The sklearn documentation has some nice discussion of this. I also have an open question about machine learning (particularly neural network) overconfidence. If your model has poor calibration, then it isn't really reasonable to claim that $$\hat y_i= p$$ means that $$P(Y=1) = p$$, since the model is, in some sense, not telling the truth.
• As an additional point (+1), it's also possible that $\hat y$ is not in $[0,1]$ and so can't be a probability. You might argue that something of this sort shouldn't be called a binary response model, I suppose Commented Sep 11, 2023 at 6:15