# Bishop equation 3.67

Equation (3.67) in Bishop's "Pattern recognition and machine learning" is the following

$$p(t|\mathbf{x}, D) = \sum_i^Lp(t|\mathbf{x}, M_i, D)p(M_i|D) \quad (3.67)$$

where $$t$$ is a target variable, $$D$$ is a dataset, $$\mathbf{x}$$ is an input vector and $$M_i$$ is a model in a set of $$L$$ models that refers to a probability distribution over the observed data $$D$$. He writes (3.67) is derived with the sum and product rules of probability. However, when I try I don't get the same result:

$$p(t|\mathbf{x}, D) = \dfrac{p(t, \mathbf{x}, D)}{p(\mathbf{x}, D)} = \dfrac{\sum_i^L p(t, \mathbf{x}, D, M_i)}{p(\mathbf{x}, D)} = \dfrac{\sum_i^L p(t, \mathbf{x}, D, M_i)}{p(\mathbf{x}, D, M_i)} \dfrac{ p(\mathbf{x}, D, M_i)}{p(\mathbf{x}, D)} = \sum_i^L p(t| \mathbf{x}, D, M_i)p(M_i|D, \mathbf{x})$$

That is I get a factor $$p(M_i|D, \mathbf{x})$$ instead of $$p(M_i|D)$$.
What am I doing wrong?

edit: My understanding is that $$D$$ is a set of data points $$\{(\mathbf{x}_1, t_1), (\mathbf{x}_2, t_2), ..., (\mathbf{x}_N, t_N)\}$$ and that $$\mathbf{x}$$ is the N+1:th input $$\mathbf{x}_{N+1}$$ from which we want to predict what the next $$t=t_{N+1}$$ is going to be. I'm not entirely sure about this though. For what it means for probabilities to be conditioned upon input vectors, I have seen examples earlier in the book where for example it is assumed that $$t = y(\mathbf{x}, \mathbf{w}) + \epsilon$$ where $$\epsilon$$ is zero-mean Gaussian noise with precision $$\beta$$ so that $$p(t|\mathbf{x}, \mathbf{w}, \beta) = \mathcal{N}(t|y(\mathbf{x}, \mathbf{w}), \beta^{-1})$$

• Could you explain what an "input vector" is and what it might mean for probabilities to be conditioned on input vectors?
– whuber
Commented Jul 6, 2021 at 15:25
• @whuber good question, I've updated my question with my understanding of it. Commented Jul 6, 2021 at 15:52

\begin{align} p(t \vert \mathbf{x}, \mathcal{D}) &= \sum^L_{i=1}p(t, \mathcal{M}_i \vert \mathbf{x}, \mathcal{D}) \tag{marginalisation} \\ &= \sum^L_{i=1} p(t \vert \mathbf{x}, \mathcal{M_i}, \mathcal{D})p(\mathcal{M}_i \vert \mathcal{D}) \tag{3.67} \end{align}
• @DancingIceCream. Without refreshing my memory of that chapter of Bishop, I can think of two possibilities. Either the model choice $\mathcal{M}_i$ is conditionally independent of $\mathbf{x}$ given the training data set $\mathcal{D}$, meaning that $p(\mathcal{M}_i \vert \mathcal{D}, \mathbf{x}) = p(\mathcal{M}_i \vert \mathcal{D})$. In this case your derivation is correct. The other possibility is that your derivation is correct mathematically, but does not respect the semantics of the model selection procedure.[...] Commented Jul 6, 2021 at 16:33
• @DancingIceCream. [...] In that $p(\mathcal{M}_i \vert \mathcal{D}, \mathbf{x})$ may make mathematical sense, and your derivation is correct; but it may not respect standard model selection procedure to additionally condition on $\mathbf{x}$ when computing the model posterior probability $p(\mathcal{M}_i \vert \mathcal{D})$. Without having looked at whether $\mathbf{x}$ is a test-data point or validation-data point, I cannot be sure of this currently. Perhaps I will edit when I get time to revisit the chapter. Commented Jul 6, 2021 at 16:34