11
$\begingroup$

I am trying to find a measure theoretic formulation of Bayes' theorem, when used in statistical inference, Bayes' theorem is usually defined as:

$$p\left(\theta|x\right) = \frac{p\left(x|\theta\right) \cdot p\left(\theta\right)}{p\left(x\right)}$$

where:

  • $p\left(\theta|x\right)$: the posterior density of the parameter.
  • $p\left(x|\theta\right)$: the statistical model (or likelihood).
  • $p\left(\theta\right)$: the prior density of the parameter.
  • $p\left(x\right)$: the evidence.

Now how would we define Bayes' theorem in a measure theoretic way?
So, I started by defining a probability space:

$$\left(\Theta, \mathcal{F}_\Theta, \mathbb{P}_\Theta\right)$$

such that $\theta \in \Theta$.
I then defined another probability space:

$$\left(X, \mathcal{F}_X, \mathbb{P}_X\right)$$

such that $x \in X$.
From here now on I don't know what to do, the joint probability space would be:

$$\left(\Theta \times X, \mathcal{F}_\Theta \otimes \mathcal{F}_X, ?\right)$$

but I don't know what the measure should be.
Bayes' theorem should be written as follow:

$$? = \frac{? \cdot \mathbb{P}_\Theta}{\mathbb{P}_X}$$

where:

$$\mathbb{P}_X = \int_{\theta \in \Theta} ? \space \mathrm{d}\mathbb{P}_\Theta$$

but as you can see I don't know the other measures and in which probability space they reside.
I stumbled upon this thread but it was of little help and I don't know how was the following measure-theoretic generalization of Bayes' rule reached:

$${P_{\Theta |y}}(A) = \int\limits_{x \in A} {\frac{{\mathrm d{P_{\Omega |x}}}}{{\mathrm d{P_\Omega }}}(y)\mathrm d{P_\Theta }}$$

I'm self-studying measure theoretic probability and lack guidance so excuse my ignorance.

$\endgroup$
  • 1
    $\begingroup$ Bayes' Theorem is not about "prior", "posterior", "likelihood", "evidence". Bayes Theorem is about marginal and conditional probabilities. Later research mapped this theorem to the concepts you mention. $\endgroup$ – Alecos Papadopoulos Jan 10 at 11:09
21
$\begingroup$

One precise formulation of Bayes' Theorem is the following, taken verbatim from Schervish's Theory of Statistics (1995).

The conditional distribution of $\Theta$ given $X=x$ is called the posterior distribution of $\Theta$. The next theorem shows us how to calculate the posterior distribution of a parameter in the case in which there is a measure $\nu$ such that each $P_\theta \ll \nu$.

Theorem 1.31 (Bayes' theorem). Suppose that $X$ has a parametric family $\mathcal{P}_0$ of distributions with parameter space $\Omega$. Suppose that $P_\theta \ll \nu$ for all $\theta \in \Omega$, and let $f_{X\mid\Theta}(x\mid\theta)$ be the conditional density (with respect to $\nu$) of $X$ given $\Theta = \theta$. Let $\mu_\Theta$ be the prior distribution of $\Theta$. Let $\mu_{\Theta\mid X}(\cdot \mid x)$ denote the conditional distribution of $\Theta$ given $X = x$. Then $\mu_{\Theta\mid X} \ll \mu_\Theta$, a.s. with respect to the marginal of $X$, and the Radon-Nikodym derivative is $$ \tag{1} \label{1} \frac{d\mu_{\Theta\mid X}}{d\mu_\Theta}(\theta \mid x) = \frac{f_{X\mid \Theta}(x\mid \theta)}{\int_\Omega f_{X\mid\Theta}(x\mid t) \, d\mu_\Theta(t)} $$ for those $x$ such that the denominator is neither $0$ nor infinite. The prior predictive probability of the set of $x$ values such that the denominator is $0$ or infinite is $0$, hence the posterior can be defined arbitrarily for such $x$ values.


Edit 1. The setup for this theorem is as follows:

  1. There is some underlying probability space $(S, \mathcal{S}, \Pr)$ with respect to which all probabilities are computed.
  2. There is a standard Borel space $(\mathcal{X}, \mathcal{B})$ (the sample space) and a measurable map $X : S \to \mathcal{X}$ (the sample or data).
  3. There is a standard Borel space $(\Omega, \tau)$ (the parameter space) and a measurable map $\Theta : S \to \Omega$ (the parameter).
  4. The distribution of $\Theta$ is $\mu_\Theta$ (the prior distribution); this is the probability measure on $(\Omega, \tau)$ given by $\mu_\Theta(A) = \Pr(\Theta \in A)$ for all $A \in \tau$.
  5. The distribution of $X$ is $\mu_X$ (the marginal distribution mentioned in the theorem); this is the probability measure on $(\mathcal{X}, \mathcal{B})$ given by $\mu_X(B) = \Pr(X \in B)$ for all $B \in \mathcal{B}$.
  6. There is a probability kernel $P : \Omega \times \mathcal{B} \to [0, 1]$, denoted $(\theta, B) \mapsto P_\theta(B)$ which represents the conditional distribution of $X$ given $\Theta$. This means that

    • for each $B \in \mathcal{B}$, the map $\theta \mapsto P_\theta(B)$ from $\Omega$ into $[0, 1]$ is measurable,
    • $P_\theta$ is a probability measure on $(\mathcal{X}, \mathcal{B})$ for each $\theta \in \Omega$, and
    • for all $A \in \tau$ and $B \in \mathcal{B}$, $$ \Pr(\Theta \in A, X \in B) = \int_A P_\theta(B) \, d\mu_\Theta(\theta). $$

    This is the parametric family of distributions of $X$ given $\Theta$.

  7. We assume that there exists a measure $\nu$ on $(\mathcal{X}, \mathcal{B})$ such that $P_\theta \ll \nu$ for all $\theta \in \Omega$, and we choose a version $f_{X\mid\Theta}(\cdot\mid\theta)$ of the Radon-Nikodym derivative $d P_\theta / d \nu$ (strictly speaking, the guaranteed existence of this Radon-Nikodym derivative might require $\nu$ to be $\sigma$-finite). This means that $$ P_\theta(B) = \int_B f_{X\mid\Theta}(x \mid \theta) \, d\nu(x) $$ for all $B \in \mathcal{B}$. It follows that $$ \Pr(\Theta \in A, X \in B) = \int_A \int_B f_{X \mid \Theta}(x \mid \theta) \, d\nu(x) \, d\mu_\Theta(\theta) $$ for all $A \in \tau$ and $B \in \mathcal{B}$. We may assume without loss of generality (e.g., see exercise 9 in Chapter 1 of Schervish's book) that the map $(x, \theta) \mapsto f_{X\mid \Theta}(x\mid\theta)$ of $\mathcal{X}\times\Omega$ into $[0, \infty]$ is measurable. Then by Tonelli's theorem we can change the order of integration: $$ \Pr(\Theta \in A, X \in B) = \int_B \int_A f_{X \mid \Theta}(x \mid \theta) \, d\mu_\Theta(\theta) \, d\nu(x) $$ for all $A \in \tau$ and $B \in \mathcal{B}$. In particular, the marginal probability of a set $B \in \mathcal{B}$ is $$ \mu_X(B) = \Pr(X \in B) = \int_B \int_\Omega f_{X \mid \Theta}(x \mid \theta) \, d\mu_\Theta(\theta) \, d\nu(x), $$ which shows that $\mu_X \ll \nu$, with Radon-Nikodym derivative $$ \frac{d\mu_X}{d\nu} = \int_\Omega f_{X \mid \Theta}(x \mid \theta) \, d\mu_\Theta(\theta). $$
  8. There exists a probability kernel $\mu_{\Theta \mid X} : \mathcal{X} \times \tau \to [0, 1]$, denoted $(x, A) \mapsto \mu_{\Theta \mid X}(A \mid x)$, which represents the conditional distribution of $\Theta$ given $X$ (i.e., the posterior distribution). This means that
    • for each $A \in \tau$, the map $x \mapsto \mu_{\Theta \mid X}(A \mid x)$ from $\mathcal{X}$ into $[0, 1]$ is measurable,
    • $\mu_{\Theta \mid X}(\cdot \mid x)$ is a probability measure on $(\Omega, \tau)$ for each $x \in \mathcal{X}$, and
    • for all $A \in \tau$ and $B \in \mathcal{B}$, $$ \Pr(\Theta \in A, X \in B) = \int_B \mu_{\Theta \mid X}(A \mid x) \, d\mu_X(x) $$

Edit 2. Given the setup above, the proof of Bayes' theorem is relatively straightforward.

Proof. Following Schervish, let $$ C_0 = \left\{x \in \mathcal{X} : \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) = 0\right\} $$ and $$ C_\infty = \left\{x \in \mathcal{X} : \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) = \infty\right\} $$ (these are the sets of potentially problematic $x$ values for the denominator of the right-hand-side of \eqref{1}). We have $$ \mu_X(C_0) = \Pr(X \in C_0) = \int_{C_0} \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) \, d\nu(x) = 0, $$ and $$ \mu_X(C_\infty) = \int_{C_\infty} \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) \, d\nu(x) = \begin{cases} \infty, & \text{if $\nu(C_\infty) > 0$,} \\ 0, & \text{if $\nu(C_\infty) = 0$.} \end{cases} $$ Since $\mu_X(C_\infty) = \infty$ is impossible ($\mu_X$ is a probability measure), it follows that $\nu(C_\infty) = 0$, whence $\mu_X(C_\infty) = 0$ as well. Thus, $\mu_X(C_0 \cup C_\infty) = 0$, so the set of all $x \in \mathcal{X}$ such that the denominator of the right-hand-side of \eqref{1} is zero or infinite has zero marginal probability.

Next, consider that, if $A \in \tau$ and $B \in \mathcal{B}$, then $$ \Pr(\Theta \in A, X \in B) = \int_B \int_A f_{X \mid \Theta}(x \mid \theta) \, d\mu_\Theta(\theta) \, d\nu(x) $$ and simultaneously $$ \begin{aligned} \Pr(\Theta \in A, X \in B) &= \int_B \mu_{\Theta \mid X}(A \mid x) \, d\mu_X(x) \\ &= \int_B \left( \mu_{\Theta \mid X}(A \mid x) \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) \right) \, d\nu(x). \end{aligned} $$ It follows that $$ \mu_{\Theta \mid X}(A \mid x) \int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t) = \int_A f_{X \mid \Theta}(x \mid \theta) \, d\mu_\Theta(\theta) $$ for all $A \in \tau$ and $\nu$-a.e. $x \in \mathcal{X}$, and hence $$ \mu_{\Theta \mid X}(A \mid x) = \int_A \frac{f_{X \mid \Theta}(x \mid \theta)}{\int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t)} \, d\mu_\Theta(\theta) $$ for all $A \in \tau$ and $\mu_X$-a.e. $x \in \mathcal{X}$. Thus, for $\mu_X$-a.e. $x \in \mathcal{X}$, $\mu_{\Theta\mid X}(\cdot \mid x) \ll \mu_\Theta$, and the Radon-Nikodym derivative is $$ \frac{d\mu_{\Theta \mid X}}{d \mu_\Theta}(\theta \mid x) = \frac{f_{X \mid \Theta}(x \mid \theta)}{\int_\Omega f_{X \mid \Theta}(x \mid t) \, d\mu_\Theta(t)}, $$ as claimed, completing the proof.


Lastly, how do we reconcile the colloquial version of Bayes' theorem found so commonly in statistics/machine learning literature, namely, $$ \tag{2} \label{2} p(\theta \mid x) = \frac{p(\theta) p(x \mid \theta)}{p(x)}, $$ with \eqref{1}?

On the one hand, the left-hand-side of \eqref{2} is supposed to represent a density of the conditional distribution of $\Theta$ given $X$ with respect to some unspecified dominating measure on the parameter space. In fact, none of the dominating measures for the four different densities in \eqref{2} (all named $p$) are explicitly mentioned.

On the other hand, the left-hand-side of \eqref{1} is the density of the conditional distribution of $\Theta$ given $X$ with respect to the prior distribution.

If, in addition, the prior distribution $\mu_\Theta$ has a density $f_\Theta$ with respect to some (let's say $\sigma$-finite) measure $\lambda$ on the parameter space $\Omega$, then $\mu_{\Theta \mid X}(\cdot\mid x)$ is also absolutely continuous with respect to $\lambda$ for $\mu_X$-a.e. $x \in \mathcal{X}$, and if $f_{\Theta \mid X}$ represents a version of the Radon-Nikodym derivative $d\mu_{\Theta\mid X}/d\lambda$, then \eqref{1} yields $$ \begin{aligned} f_{\Theta \mid X}(\theta \mid x) &= \frac{d \mu_{\Theta \mid X}}{d\lambda}(\theta \mid x) \\ &= \frac{d \mu_{\Theta \mid X}}{d\mu_\Theta}(\theta \mid x) \frac{d \mu_{\Theta}}{d\lambda}(\theta) \\ &= \frac{d \mu_{\Theta \mid X}}{d\mu_\Theta}(\theta \mid x) f_\Theta(\theta) \\ &= \frac{f_\Theta(\theta) f_{X\mid \Theta}(x\mid \theta)}{\int_\Omega f_{X\mid\Theta}(x\mid t) \, d\mu_\Theta(t)} \\ &= \frac{f_\Theta(\theta) f_{X\mid \Theta}(x\mid \theta)}{\int_\Omega f_\Theta(t) f_{X\mid\Theta}(x\mid t) \, d\lambda(t)}. \end{aligned} $$ The translation between this new form and \eqref{2} is $$ \begin{aligned} p(\theta \mid x) &= f_{\Theta \mid X}(\theta \mid x) = \frac{d \mu_{\Theta \mid X}}{d\lambda}(\theta \mid x), &&\text{(posterior)}\\ p(\theta) &= f_\Theta(\theta) = \frac{d \mu_\Theta}{d\lambda}(\theta), &&\text{(prior)} \\ p(x \mid \theta) &= f_{X\mid\Theta}(x\mid\theta) = \frac{d P_\theta}{d\nu}(x), &&\text{(likelihood)} \\ p(x) &= \int_\Omega f_\Theta(t) f_{X\mid\Theta}(x\mid t) \, d\lambda(t). &&\text{(evidence)} \end{aligned} $$

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Why should $\Omega$ be a Borel space instead of some other measure space? $\endgroup$ – Dave Jan 9 at 22:53
  • 4
    $\begingroup$ @Dave Borel spaces are easier to work with for technical reasons, while also being fairly general. For example, conditional distributions of random variables taking values in a Borel space always exist, whereas they might not exist for random variables taking values in a non-Borel space. Fortunately, most spaces in practice are Borel spaces. For example, every Borel subset of a complete, separable metric space is a Borel space. $\endgroup$ – Artem Mavrin Jan 9 at 23:11
  • 1
    $\begingroup$ I just checked the edit, your answer is extremely clear and detailed and helped me a lot, thank you very much for the time and effort @ArtemMavrin. $\endgroup$ – Blg Khalil Jan 10 at 0:14
  • 1
    $\begingroup$ @BlgKhalil yes, you could call it that if you want, and it would be more consistent with the rest of the notation $\endgroup$ – Artem Mavrin Jan 11 at 21:16
  • 1
    $\begingroup$ @BlgKhalil glad to help :) $\endgroup$ – Artem Mavrin Jan 11 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.