Controlling false positive rate and false negative rate

Question

My notes define the false positive rate as

$$\dfrac{\text{false positives}}{\text{true negatives} + \text{false positives}} = \dfrac{\text{false positives}}{\text{total negatives}}$$

and the false negative rate as

$$\dfrac{\text{false negatives}}{\text{false negatives} + \text{true positives}} = \dfrac{\text{false negatives}}{\text{total positives}}$$

It then says the following:

We typically cannot control both these errors:

• If we change parameters in the classifier that make one smaller, then the other one often gets larger.

I don't really see why this must mathematically be true. It seems to me that these two rates depend on the efficacy of the technology, and that there is no mathematical reason why both cannot be improved at the same time. Am I misunderstanding something, or is what is stated in the notes incorrect?

Answer 1

You are misunderstanding. The statement is standard—I've said a version of it myself many times. It applies to a given dataset and a specific model of it. Certainly, with better data, more informative variables, and a better model, you can do better on both metrics. If those are available to you, then have at it. In the end though, you have what you have. At that point, if you wanted to convert the model's output (say, a predicted probability) into a predicted category, you would need to compare the outputted value for an observation to a threshold and give that observation a thumbs up or thumbs down. Within those constraints, you can improve your false negative rate (or your false positive rate) by changing the threshold, but the other will get worse. That is, if you want predicted categories (and you don't necessarily have to get them) you will face a trade off between those two rates. The ROC curve shows you the set of trade offs available to you.