Is it possible to decompose the model evidence?

Question

Assume I want to apply Bayes theorem with some state variable $x$ (scalar or vector, doesn't matter) and an observation vector $\mathbf{y}=[y_1,...,y_N]^T$::

$$p(x|\mathbf{y})=\frac{p(x)p(\mathbf{y}|x)}{p(\mathbf{y})}$$

Further assume that my likelihood is composite, i.e.:

$$p(\mathbf{y}|x)=\prod_{i=1}^N p(y_i|x)$$

where the $y_i$ are independent observations. Would it be possible to also decompose the model evidence $p(\mathbf{y})$ into its components as:

$$p(x|\mathbf{y})=\frac{p(x)p(\mathbf{y}|x)}{p(y_1)p(y_2)...p(y_N)}$$

If not, why?

Answer 1

We can obtain some intuition from an example.

Let $p(y_i|x) = \textsf{N}(y_i|x,1)$, $p(x) = \textsf{N}(x|0,s^2)$, and let $N = 2$. Then \begin{equation} p(y) = \textsf{N}(y|0_2,\Sigma) \end{equation} where $y = (y_1,y_2)$, $0_2 = (0,0)$, and \begin{equation} \Sigma = \begin{pmatrix} s^2 + 1 & s^2 \\ s^2 & s^2 + 1 \end{pmatrix} . \end{equation} The point is that variation in $x$ induces dependence among the $y_i$.

In this example, we can eliminate the variation in $x$ by letting $s$ go to zero, thereby eliminating the dependence among the $y_i$: \begin{equation} \lim_{s\to 0}\ \Sigma = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} . \end{equation} In the limit, $p(y) = p(y_1)\,p(y_2)$. The limiting case amounts to putting a point-mass prior on $x$.

Answer 2

You can, yes. In the more general case, if we also expand the numerator in the same way, we'd get the following:

$$p(x|\mathbf{y}) = p(x) \frac{p(y_1|x)}{p(y_1)} \frac{p(y_2|x, y_1)}{p(y_2|y_1)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})},$$

where the conditioning on previous $y$ in the numerator and the denominator vanishes when the observations are independent, which is the case you're looking at.

This decomposes the likelihood/marginal ratio into a ratio for each observation, conditional on previous ones. Sticking with the general case, if we then collapse these terms one by one...

$$\begin{align*} p(x|\mathbf{y}) &= p(x|y_1) \frac{p(y_2|x, y_1)}{p(y_2|y_1)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})}\\ &= p(x|y_1, y_2) \frac{p(y_3|x, y_1, y_2)}{p(y_3|y_1, y_2)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})}\ldots\\ &= p(x|y_1, y_2, \ldots, y_N), \end{align*}$$

we can see that this is equivalent to finding the posterior conditional on $y_1$, then using this as the prior for observing $y_2$, and so on. Hence, we can decompose the model evidence as if we'd only taken one observation before each model update.