bias-variance tradeoff vs precision and recall

Question

Can anyone explain the link between bias-variance tradeoff and precision-recall tradeoff. Are they effectively the same thing?

Answer 1

Well there are parallels between the two, for the mean squared error case, the error of the model is due to its bias and its variance.

$MSE(W) = Bias^2(W) + Var(W)$

where, $Bias(W) = E[W] - \theta$, if $\theta$ is the true parameter.

If the model is able to fit the training dataset very well it would have a low bias. But that is not necessarily a good thing, as it could have very high variance if it is a very high dimensional model or has tons of parameters. Essentially the model is just "memorizing" the data with its parameters instead of generalizing from it. On the other hand, a less powerful model might not do so well on the training data but it generalizes better. Such a model would have a higher bias and lower variance.

Now moving on to precision and recall, which are related to minimizing false positives and false negatives respectively. In the extreme case, you could have a classifier which simply remembers the training set, in this case you would have a recall close to or even equal to $1$ and a precision close to $0$. A high recall and low precision model corresponds to the case of having high variance and low bias. Similarly you could have a model which gets some false negatives but gets fewer false positives, ie, it is high precision - low recall, then it corresponds to the high bias - low variance case.

However, you need to strike the right balance in both cases. The goal is to reduce the total error in the regression case and to simultaneously increase both precision and recall in the classification scenario (F-score). It is a mistake to focus on optimizing the accuracy instead of the F-score as the classification accuracy is a biased measure for skewed distribution of classes.

Answer 2

In general precision-recall tradeoff is seen as a discrimination threshold for what we consider positive. A more strict/picky/pessimistic threshold will lead to higher precision (at the cost of ignoring possible positive cases). A more lax/optimistic threshold will lead to higher recall (at the cost of false alarms).

Bias-Variance has more to do with model complexity and doesn't have to be limited to the framework of classification tasks. A model that is more simple (or more biased) will still have to decide how to threshold between the positive and negative class, as will a more complex model (or more variance).

While the trade-offs feel similar they are a bit different. The bias-variance trade-off is more concrete if you're looking at error in the empirical risk minimization framework: The expected error = irreducible error + bias^2 + variance. There is no such equation for precision-recall that clearly delineates this trade-off as far as I am aware. I believe in general it is more of an observed thing than "proved" thing.

The way I like to think about it is, precision/recall both focus on the positive class. Their formulas wrt the contingency table respectively is TP/P_est = TP/(TP + FP) and TP/P = TP/(TP + FN). Literally everything about them is the same except the denominator. Precision will penalize a FP, recall will penalize a FN. Because they both focus on the positive class only, it's like they are on a seesaw with each other for the positive class.

Note: TP = true positive. FP = false positive. FN = false negative. P_est = total estimated/predicted positives. P = total actual positives.