Skip to main content
Added remark on error rate balance != sufficiency
Source Link
Eike P.
  • 3.1k
  • 13
  • 29

The essential intuition for why calibration by groups and separation (=balance for the positive/negative classes) are incompatible is that the average score of a calibrated classifier within each group is equal to the base rate of that group, $p(y|x, \text{group}=i)$. From this, it is already almost apparent that equal average risk scores in the positive/negative classes of each group cannot be achieved, if there are base rate differences (and the classifier is not perfect).

More formally, from the above insight, one can derive that calibrated classifiers lie on straight lines in the following diagram for the different groups, and one would need the lines for the different groups to intersect - which can only happen if the classifier is either perfect or if there are no base rate differences between the groups.

enter image description here

Crucially, error rate balance (=equal TPR, FPR) is not the same as separation, and it is in principle possible to achieve error rate balance and calibration by group at the same time. (I provide an example of this on my blog, see link below.) To achieve exact error rate balance, the ROC curves for the different groups would have to intersect, however, which is unlikely to be the case for any practical application.

I just summarized all of this in more detail in a post on my personal blog.

The essential intuition is that the average score of a calibrated classifier within each group is equal to the base rate of that group, $p(y|x, \text{group}=i)$. From this, it is already almost apparent that equal average risk scores in the positive/negative classes of each group cannot be achieved, if there are base rate differences (and the classifier is not perfect).

More formally, from the above insight, one can derive that calibrated classifiers lie on straight lines in the following diagram for the different groups, and one would need the lines for the different groups to intersect - which can only happen if the classifier is either perfect or if there are no base rate differences between the groups.

enter image description here

I just summarized all of this in more detail in a post on my personal blog.

The essential intuition for why calibration by groups and separation (=balance for the positive/negative classes) are incompatible is that the average score of a calibrated classifier within each group is equal to the base rate of that group, $p(y|x, \text{group}=i)$. From this, it is already almost apparent that equal average risk scores in the positive/negative classes of each group cannot be achieved, if there are base rate differences (and the classifier is not perfect).

More formally, from the above insight, one can derive that calibrated classifiers lie on straight lines in the following diagram for the different groups, and one would need the lines for the different groups to intersect - which can only happen if the classifier is either perfect or if there are no base rate differences between the groups.

enter image description here

Crucially, error rate balance (=equal TPR, FPR) is not the same as separation, and it is in principle possible to achieve error rate balance and calibration by group at the same time. (I provide an example of this on my blog, see link below.) To achieve exact error rate balance, the ROC curves for the different groups would have to intersect, however, which is unlikely to be the case for any practical application.

I just summarized all of this in more detail in a post on my personal blog.

Source Link
Eike P.
  • 3.1k
  • 13
  • 29

The essential intuition is that the average score of a calibrated classifier within each group is equal to the base rate of that group, $p(y|x, \text{group}=i)$. From this, it is already almost apparent that equal average risk scores in the positive/negative classes of each group cannot be achieved, if there are base rate differences (and the classifier is not perfect).

More formally, from the above insight, one can derive that calibrated classifiers lie on straight lines in the following diagram for the different groups, and one would need the lines for the different groups to intersect - which can only happen if the classifier is either perfect or if there are no base rate differences between the groups.

enter image description here

I just summarized all of this in more detail in a post on my personal blog.