There are several results in the literature now, stating that a classifier cannot fulfill calibration and error rate balance at the same time if there are actual differences between groups. To pick one exemplary result, Kleinberg et al. (2016) derive that the following three conditions can only be fulfilled simultaneously if there are no actual differences between the groups (copying verbatim from their paper in the following):
- Calibration within groups, i.e., for each group t, and each bin b with associated score vb, the expected number of people from group t in b who belong to the positive class should be a vb fraction of the expected number of people from group t assigned to b.
- Balance for the negative class, i.e., the average score assigned to people of group 1 who belong to the negative class should be the same as the average score assigned to people of group 2 who belong to the negative class.
- Balance for the positive class, i.e., the average score assigned to people of group 1 who belong to the positive class should be the same as the average score assigned to people of group 2 who belong to the positive class.
I can follow their derivation (and other, similar ones), but I am still missing an intuition for why I cannot have a well-calibrated classifier that achieves error rate balance in any non-trivial case? Why are these two requirements contradictory? It seems to have something to do with the fact that if the base rates differ between groups, one cannot have true positive rate, false positive rate, positive predictive value, and negative predictive value all be equal across groups. (See the Fair ML book, p. 56-57.) But it still can't wrap my head around why - intuitively - that is not possible. Maybe someone has a nice illustrative example or can otherwise provide intuition?