# Lack of rigor when describing prediction metrics

I constantly see metrics that measure the quality of a classifier's predictions, such as TPR, FPR, Precision, etc., being described as probabilities (see Wikipedia, for example).

As far as I know, these are not probabilities but observations of a classifier's predictive behavior estimated during testing: for instance, they measure the rate of observed True Positives, False Positives, and so on.

Wouldn't it be more correct to distinguish these from the actual probabilities? That is, to say that by the Law of Large Numbers, these mesaurements converge to the probabilities. For instance, Precision (the measurement) converges to the Bayesian detection Rate (the actual probability).

• There are numerous such statistics based on confusion matrices. Some of them are probabilities, and some are not. Commented Jan 8 at 14:22

These strike me as perfectly reasonable estimates of the parameter of a Bernoulli distribution, which we typically interpret as a probability.

Probability that a classification will be correct (accuracy)

Probability that a positive case will be detected (sensitivity)

Probability that a case asserted to be positive really will turn out to be (precision)

If you want to say that these are just estimates of true probability values (the true Bernoulli parameters), that's fine but also not controversial. We should expect the performance in production to differ at least slightly from what we observe in development.

• 🎯 Exactly (+1), some of the statistics computed from a confusion matrix are conditional probabilities. Commented Jan 8 at 14:24
• yes, I agree they are estimates of probabilities, but they are not the true probabilities. My point is precisely that the distinction is important but not sufficiently acknowledged. They say "TPR is the probability that..." when they should say "TPR is an estimate of the probability that..." Commented Jan 9 at 11:38
• @synack: they are a true probability: the probability in the validation set. There is some laziness in not calling them estimates of the value of interest (probability of full population) or not always giving confidence intervals, but this is not the biggest statistical crime of the machine learning community :). Also in many cases, the validation set has millions of rows so the standard error is unlikely to actually be meaningful. Commented Jan 9 at 17:28
• I've referred to such practices as "slang" a number of times and stand by doing so.
– Dave
Commented Jan 9 at 17:31