Measuring response bias from a confusion matrix Given a confusion matrix of a binary classifier, what are the best measures of response bias towards one of the classes?
One idea that comes to mind is Signal Detection Theory's criterion, but this measure assumes a Gaussian noise model.
Is there a more straightforward, well-tested measure of response bias? For example, we could divide the number of predicted positives (TP + FP) by the number of positives (TP + FN), but I'm not sure that such a ratio would be "well behaved".
 A: The Python package sdt_metrics by Roger Lew implements several non-parametric response bias measures. Unfortunately, the package is not maintained, but the references are still useful.
One of these references is an empirical study comparing five response bias measures:

See, J. E., Warm, J. S., Dember, W. N., and Howe, S. R. (1997).
Vigilance and signal detection theory: An empirical evaluation of five
measures of response bias. https://doi.org/10.1518/001872097778940704

Among the parametric response bias measures, they recommend B"D:
$$B’’_D = \frac{(1-H)(1-FA)-(H)(FA)}{(1-H)(1-FA)+(H)(FA)}$$
Note that care must be taken when $H$ (hit rate) or $FA$ (false alarm rate) are at their boundaries:

Hautus, M.J. Corrections for extreme proportions and their biasing
effects on estimated values of d′. Behavior Research Methods,
Instruments, & Computers 27, 46–51 (1995).
https://doi.org/10.3758/BF03203619

A: I'll expand on my comment here. This answer was partially adapted from this one.
Let $Y$ represent a class label and $\hat Y$ represent a class prediction. In the binary case, let the two possible values for $Y$ and $\hat Y$ be $0$ and $1$, which represent the classes. Next, suppose that the confusion matrix for $Y$ and $\hat Y$ is:





$\hat Y = 0$
$\hat Y = 1$




$Y = 0$
10
20


$Y = 1$
30
40




With hindsight, let us normalize the rows and columns of this confusion matrix, such that the sum of all elements of the confusion matrix is $1$. Currently, the sum of all elements of the confusion matrix is $10 + 20 + 30 + 40 = 100$, which is our normalization factor. After dividing the elements of the confusion matrix by the normalization factor, we get the following normalized confusion matrix:





$\hat Y = 0$
$\hat Y = 1$




$Y = 0$
$\frac{1}{10}$
$\frac{2}{10}$


$Y = 1$
$\frac{3}{10}$
$\frac{4}{10}$




With this formulation of the confusion matrix, we can interpret it as an estimate of the joint probability mass function (PMF) of $Y$ and $\hat Y$. This interpretation allows us to measure how often a certain class is predicted. For example, suppose we wanted to compute $P(\hat{Y} = 1)$, which is how often the classifier predicts class $1$. Using the law of total probability, we can compute this as $P(\hat{Y} = 1) = P(\hat{Y} = 1,Y = 1) + P(\hat{Y} = 1,Y = 0) = \frac{4}{10} + \frac{2}{10} = \frac{6}{10}$. Therefore, the classifier predicts class $1$ about $60\%$ of the time. Note, however, that these are all estimates.
