# Is it possible to decompose the model evidence?

Assume I want to apply Bayes theorem with some state variable $$x$$ (scalar or vector, doesn't matter) and an observation vector $$\mathbf{y}=[y_1,...,y_N]^T$$::

$$p(x|\mathbf{y})=\frac{p(x)p(\mathbf{y}|x)}{p(\mathbf{y})}$$

Further assume that my likelihood is composite, i.e.:

$$p(\mathbf{y}|x)=\prod_{i=1}^N p(y_i|x)$$

where the $$y_i$$ are independent observations. Would it be possible to also decompose the model evidence $$p(\mathbf{y})$$ into its components as:

$$p(x|\mathbf{y})=\frac{p(x)p(\mathbf{y}|x)}{p(y_1)p(y_2)...p(y_N)}$$

If not, why?

• The denominator is incorrect because the $y_i$'s become dependent once $x$ is integrated out. Sep 15, 2021 at 11:47

We can obtain some intuition from an example.

Let $$p(y_i|x) = \textsf{N}(y_i|x,1)$$, $$p(x) = \textsf{N}(x|0,s^2)$$, and let $$N = 2$$. Then $$$$p(y) = \textsf{N}(y|0_2,\Sigma)$$$$ where $$y = (y_1,y_2)$$, $$0_2 = (0,0)$$, and $$$$\Sigma = \begin{pmatrix} s^2 + 1 & s^2 \\ s^2 & s^2 + 1 \end{pmatrix} .$$$$ The point is that variation in $$x$$ induces dependence among the $$y_i$$.

In this example, we can eliminate the variation in $$x$$ by letting $$s$$ go to zero, thereby eliminating the dependence among the $$y_i$$: $$$$\lim_{s\to 0}\ \Sigma = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} .$$$$ In the limit, $$p(y) = p(y_1)\,p(y_2)$$. The limiting case amounts to putting a point-mass prior on $$x$$.

You can, yes. In the more general case, if we also expand the numerator in the same way, we'd get the following:

$$p(x|\mathbf{y}) = p(x) \frac{p(y_1|x)}{p(y_1)} \frac{p(y_2|x, y_1)}{p(y_2|y_1)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})},$$

where the conditioning on previous $$y$$ in the numerator and the denominator vanishes when the observations are independent, which is the case you're looking at.

This decomposes the likelihood/marginal ratio into a ratio for each observation, conditional on previous ones. Sticking with the general case, if we then collapse these terms one by one...

\begin{align*} p(x|\mathbf{y}) &= p(x|y_1) \frac{p(y_2|x, y_1)}{p(y_2|y_1)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})}\\ &= p(x|y_1, y_2) \frac{p(y_3|x, y_1, y_2)}{p(y_3|y_1, y_2)} \ldots \frac{p(y_N|x, y_1, \ldots, y_{N-1})}{p(y_N | y_1, \ldots, y_{N-1})}\ldots\\ &= p(x|y_1, y_2, \ldots, y_N), \end{align*}

we can see that this is equivalent to finding the posterior conditional on $$y_1$$, then using this as the prior for observing $$y_2$$, and so on. Hence, we can decompose the model evidence as if we'd only taken one observation before each model update.