# Does perfect calibration imply perfect accuracy? [closed]

This is a problem from cs229-autumn2018-ps2.

I am given a logistic regression model that is perfectly calibrated, as in, it satisfies the following property:

$$\frac{\Sigma_{i \in Ia,b} P(y^{(i)}=1|x^{(i)};\theta)}{|\{i \in Ia,b\}|} = \frac{\Sigma_{i \in Ia,b} \mathbf{1}\{y^{(i)}=1\}}{|\{i \in Ia,b\}|}$$

where the probability $$P$$ follows a sigmoid function, and $$I_{a,b} = \{i | i \in \{1,...,m\}, h_{\theta}(x^{(i)}) \in (a,b)\}$$.

Question 2 asks, if a binary classification model is perfectly calibrated, that is, the above equation holds for any $$(a,b) \subset [0, 1]$$, is the model perfectly accurate? What about the converse?

Link to the original question and solution:

https://github.com/maxim5/cs229-2018-autumn/blob/main/problem-sets/PS2/ps2.pdf

https://github.com/maxim5/cs229-2018-autumn/blob/main/problem-sets/PS2/PS2-2%20Model%20Calibration.ipynb

I would still like to provide my understanding. The equation, in layman's terms, states that, for all examples where our model predicts their output to have probability in the range $$(a,b)$$, the average of the predicted probabilities must equal to the percentage that is actually true. (e.g. If our model predicts 100 examples to have probability [0.5, 0.6], and the average is 51%, among these 100 examples, exactly 51 should be true.)

Now, for question 2, the solution gives an example where all the outputs are true, $$\{x^{(i)}, y^{(i)}=1\}_{i=1}^{m}$$, and takes range $$(0.5, 1)$$. It states:

$$\frac{\Sigma_{i \in Ia,b} \mathbf{1}\{y^{(i)}=1\}}{|\{i \in Ia,b\}|} = \frac{\Sigma_{i \in Ia,b} P(y^{(i)}=1|x^{(i)};\theta)}{|\{i \in Ia,b\}|} < 1$$.

I am having a hard time understanding why. I could only guess that a sigmoid function can never return 1, unless the exponent is negative infinity, which is impossible.

For the converese, perfect accuracy does not imply perfect calibration, the solution gives:

$$\frac{\Sigma_{i \in Ia,b} \mathbf{1}\{y^{(i)}=1\}}{|\{i \in Ia,b\}|} = 1 > \frac{\Sigma_{i \in Ia,b} P(y^{(i)}=1|x^{(i)};\theta)}{|\{i \in Ia,b\}|}$$.

Much appreciated.