Skip to main content
2 of 2
add brier score example
ischmidt20
  • 471
  • 2
  • 9

In statistics (and to a lesser extent ML), we would use "uncertainty" to refer to uncertainty in the model itself (e.g. variability in model terms resulting from sampling variation). The term we would use to describe a maximum-entropy prediction is irreducible error. These two terms, together with model bias, sum up to the total expected generalization error, which is how much error we expect any given model to have when it is applied to out-of-sample data.

For example, while our prediction might assign equal weight to each of 10 different classes, "uncertainty" would refer to the fact that if we had sampled a different training set, we might get a prediction that puts 11% on half of the classes and 9% on the other half. The fact that in either case we are so "uncertain" about which class is the correct one would result from the irreducible error (or perhaps model bias, if the "true" probabilities were more skewed and the model was simply incapable of discerning such relationships).

Edit: This gets more confusing because when making probabilistic predictions, because in these cases we can decompose our "score" (loss function) into a different set of three terms: reliability, resolution, and uncertainty. In this case, uncertainty is equivalent to the "irreducible error" I described above.

So I guess you could argue that "uncertainty" is just as valid to use in an information theory context. So I suppose that which term you use depends on your audience and their collective background. "Uncertainty" is just another word in a long list of terms that mean different things to different people...

ischmidt20
  • 471
  • 2
  • 9