The interpretation of the term "uncertainty" in statistics vs. information theory vs. machine learning

Question

I have an ensemble model consisting from multiple classifiers and I wish to quantify the uncertainty of the predictions the ensemble model makes. From an information theory / machine learning perspective, entropy can be used to quantify the degree of prediction confidence which is minimized when all classifications are equally likely, say we have 10 output classes and each individual model in the ensemble of 10 models votes for a different class from all the others, and is maximized when all individual models vote for the same class.

In this setting, entropy quantifies well the "uncertainty" of the prediction. A colleague argued that the use of the term "uncertainty" is wrong in this modeling context as it is usually used to mean confidence intervals or variance of a prediction.

My question is: is this argument justified? Is the term "uncertainty" reserved more for statistics or is it equally valid to use it in information theory perspective? If the term "uncertainty" is reserved more for statistics, then what would be the correct term to use in the information theory (and/or machine learning) context?

Answer 1

In statistics (and to a lesser extent ML), we would use "uncertainty" to refer to uncertainty in the model itself (e.g. variability in model terms resulting from sampling variation). The term we would use to describe a maximum-entropy prediction is irreducible error. These two terms, together with model bias, sum up to the total expected generalization error, which is how much error we expect any given model to have when it is applied to out-of-sample data.

For example, while our prediction might assign equal weight to each of 10 different classes, "uncertainty" would refer to the fact that if we had sampled a different training set, we might get a prediction that puts 11% on half of the classes and 9% on the other half. The fact that in either case we are so "uncertain" about which class is the correct one would result from the irreducible error (or perhaps model bias, if the "true" probabilities were more skewed and the model was simply incapable of discerning such relationships).

Edit: This gets more confusing because when making probabilistic predictions, because in these cases we can decompose our "score" (loss function) into a different set of three terms: reliability, resolution, and uncertainty. In this case, uncertainty is equivalent to the "irreducible error" I described above.

So I guess you could argue that "uncertainty" is just as valid to use in an information theory context. So I suppose that which term you use depends on your audience and their collective background. "Uncertainty" is just another word in a long list of terms that mean different things to different people...