Aleatoric and Epistemic Uncertainty in the Framework of Bayesian and Frequentist

Question

Beginning with definitions of Aleatoric and Epistemic Uncertainty from this paper:

Aleatoric: Aleatoric uncertainty refers to the intrinsic uncertainty of a particular system and the observed data. It arises due to the intrinsic and irreducible stochastic variability in the data-generating process. Aleatoric uncertainty — or data uncertainty — cannot be readily reduced as it is inherent to the measurement data.

Epistemic: Epistemic uncertainty — or model uncertainty — arises from our ignorance about the underlying physical process itself reflecting our lack of knowledge about its structure or its parameters. In machine learning, epistemic uncertainty is associated with model structure.

I am wondering about what (or if there is) the relationship between Epistemic and Aleatoric Uncertainty to Bayesian and the Frequentist paradigm is. From the Machine Learning book by Bishop section 1.2.3:

"In both the Bayesian and frequentist paradigms, the likelihood function p(D|w) plays a central role. However, the manner in which it is used is fundamentally different in the two approaches. In a frequentist setting, w is considered to be a fixed parameter, whose value is determined by some form of ‘estimator’, and error bars on this estimate are obtained by considering the distribution of possible data sets D. By contrast, from the Bayesian viewpoint there is only a single data set D (namely the one that is actually observed), and the uncertainty in the parameters is expressed through a probability distribution over w."

So is epistemic uncertainty Bayesian and aleatoric uncertainty frequentist? Or another question how does a Bayesian represent aleatoric uncertainty? Is it all somehow captured in the posterior distribution of the weight parameters?

Answer 1

If this answer is helpful then you might also be interested in some answers to related questions on this site (see e.g., here, here, here, here and here)

You can find out more about this by reading literature on philosophy relating to probability. (Some introductory discussion and references are available here.) The first thing to note on this subject is that the probability operator can be interpreted epistemically (i.e., referring to a measurement of our own certainty/uncertainty) or it can be posited that it corresponds to some actual metaphysical reality (i.e., that there is some "chance" property of reality that is well-described by the mathematics of probability). The metaphysical/aleatory interpretation is speculative, whereas the epistemic interpretation can be validated by showing that the mathematics of probability amounts to a rational way to deal with our own uncertainty about reality. In any application of probability it is possible to adopt either interpretation, though the aleatory interpretation is speculative and therefore often avoided. The epistemic interpretation is popular precisely because it can be definitively validated as a reasoning tool, and so it bypasses metaphysical speculations on determinism, indeterminism, etc.

In the Bayesian paradigm, the "standard" approach is to use the epistemic interpretation of probability and therefore view the field as one that tells people how to reason rationally under uncertainty (without making any claim that there is probability in nature). This approach is the one taken in Bernardo and Smith (1994) (see esp. Ch 1-2) and it is probably the dominant way that Bayesian practitioners view their approach. Some Bayesian practitioners may believe that there is also an inherent property in reality that corresponds to the mathematics of probability, but this is a more marginal view and it is not necessary to apply the paradigm. Most Bayesian practitioners avoid this interpretation because it is not necessary to apply the paradigm, and because moving into that interpretation raises difficult speculations on determinism, indeterminism, etc.

You also ask about the "frequentist" paradigm, which is a bit of a tricky area, since terminology here is often overly expansive. If you begin with the frequentist interpretation of probability, this is one that is framed purely as a mathematical definition relating to an infinite series of events: probability is defined as being equivalent to the long-run frequency of an event in an infinite sequence of repeatable trials. If you are willing to accept the existence of an infinite sequence of trials in reality (which is itself philosophically controversial) then this interpretation is aleatory in nature. Unlike other aleatory interpretations, this interpretation does not require speculation on determinism, indeterminism, etc., since it frames probability as a deterministic function of the indicators for a sequence of events. As I have said in several of the linked posts, all Bayesians are also "frequentists", in the sense that we accept the laws of large numbers and agree that probability corresponds to limiting relative frequency in appropriate circumstances. If you accept that there is such a thing as an infinite sequence of events then there is also a limiting relative frequency of the indicators for those events and the laws of large numbers apply under appropriate circumstances. All of this is purely mathematical and has little philosophical import. If you take an epistemic view of probability then you will merely acknowledge that epistemic probabilities of events correspond to limiting relative frequency under appropriate conditions. If you take an aleatory view of probability then you will likewise acknowledge that these aleatory probabilities of events correspond to limiting relative frequency under appropriate conditions.