3
$\begingroup$

Can anyone explain the link between bias-variance tradeoff and precision-recall tradeoff. Are they effectively the same thing?

$\endgroup$
1
$\begingroup$

Well there are parallels between the two, for the mean squared error case, the error of the model is due to its bias and its variance.

$MSE(W) = Bias^2(W) + Var(W)$

where, $Bias(W) = E[W] - \theta$, if $\theta$ is the true parameter.

If the model is able to fit the training dataset very well it would have a low bias. But that is not necessarily a good thing, as it could have very high variance if it is a very high dimensional model or has tons of parameters. Essentially the model is just "memorizing" the data with its parameters instead of generalizing from it. On the other hand, a less powerful model might not do so well on the training data but it generalizes better. Such a model would have a higher bias and lower variance.

Now moving on to precision and recall, which are related to minimizing false positives and false negatives respectively. In the extreme case, you could have a classifier which simply remembers the training set, in this case you would have a recall close to or even equal to $1$ and a precision close to $0$. A high recall and low precision model corresponds to the case of having high variance and low bias. Similarly you could have a model which gets some false negatives but gets fewer false positives, ie, it is high precision - low recall, then it corresponds to the high bias - low variance case.

However, you need to strike the right balance in both cases. The goal is to reduce the total error in the regression case and to simultaneously increase both precision and recall in the classification scenario (F-score). It is a mistake to focus on optimizing the accuracy instead of the F-score as the classification accuracy is a biased measure for skewed distribution of classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.