1
$\begingroup$

Equation (3.67) in Bishop's "Pattern recognition and machine learning" is the following

$$p(t|\mathbf{x}, D) = \sum_i^Lp(t|\mathbf{x}, M_i, D)p(M_i|D) \quad (3.67)$$

where $t$ is a target variable, $D$ is a dataset, $\mathbf{x}$ is an input vector and $M_i$ is a model in a set of $L$ models that refers to a probability distribution over the observed data $D$. He writes (3.67) is derived with the sum and product rules of probability. However, when I try I don't get the same result:

$$p(t|\mathbf{x}, D) = \dfrac{p(t, \mathbf{x}, D)}{p(\mathbf{x}, D)} = \dfrac{\sum_i^L p(t, \mathbf{x}, D, M_i)}{p(\mathbf{x}, D)} = \dfrac{\sum_i^L p(t, \mathbf{x}, D, M_i)}{p(\mathbf{x}, D, M_i)} \dfrac{ p(\mathbf{x}, D, M_i)}{p(\mathbf{x}, D)} = \sum_i^L p(t| \mathbf{x}, D, M_i)p(M_i|D, \mathbf{x})$$

That is I get a factor $p(M_i|D, \mathbf{x})$ instead of $p(M_i|D)$.
What am I doing wrong?

edit: My understanding is that $D$ is a set of data points $\{(\mathbf{x}_1, t_1), (\mathbf{x}_2, t_2), ..., (\mathbf{x}_N, t_N)\}$ and that $\mathbf{x}$ is the N+1:th input $\mathbf{x}_{N+1}$ from which we want to predict what the next $t=t_{N+1}$ is going to be. I'm not entirely sure about this though. For what it means for probabilities to be conditioned upon input vectors, I have seen examples earlier in the book where for example it is assumed that $$t = y(\mathbf{x}, \mathbf{w}) + \epsilon$$ where $\epsilon$ is zero-mean Gaussian noise with precision $\beta$ so that $$p(t|\mathbf{x}, \mathbf{w}, \beta) = \mathcal{N}(t|y(\mathbf{x}, \mathbf{w}), \beta^{-1})$$

$\endgroup$
2
  • $\begingroup$ Could you explain what an "input vector" is and what it might mean for probabilities to be conditioned on input vectors? $\endgroup$
    – whuber
    Commented Jul 6, 2021 at 15:25
  • $\begingroup$ @whuber good question, I've updated my question with my understanding of it. $\endgroup$ Commented Jul 6, 2021 at 15:52

1 Answer 1

0
$\begingroup$

\begin{align} p(t \vert \mathbf{x}, \mathcal{D}) &= \sum^L_{i=1}p(t, \mathcal{M}_i \vert \mathbf{x}, \mathcal{D}) \tag{marginalisation} \\ &= \sum^L_{i=1} p(t \vert \mathbf{x}, \mathcal{M_i}, \mathcal{D})p(\mathcal{M}_i \vert \mathcal{D}) \tag{3.67} \end{align}

$\endgroup$
3
  • $\begingroup$ Thanks, but I don't understand the last equality. If I employ the algebra I show in my post I won't get the same answer as you. $\endgroup$ Commented Jul 6, 2021 at 15:54
  • $\begingroup$ @DancingIceCream. Without refreshing my memory of that chapter of Bishop, I can think of two possibilities. Either the model choice $\mathcal{M}_i$ is conditionally independent of $\mathbf{x}$ given the training data set $\mathcal{D}$, meaning that $p(\mathcal{M}_i \vert \mathcal{D}, \mathbf{x}) = p(\mathcal{M}_i \vert \mathcal{D})$. In this case your derivation is correct. The other possibility is that your derivation is correct mathematically, but does not respect the semantics of the model selection procedure.[...] $\endgroup$
    – microhaus
    Commented Jul 6, 2021 at 16:33
  • 1
    $\begingroup$ @DancingIceCream. [...] In that $p(\mathcal{M}_i \vert \mathcal{D}, \mathbf{x})$ may make mathematical sense, and your derivation is correct; but it may not respect standard model selection procedure to additionally condition on $\mathbf{x}$ when computing the model posterior probability $p(\mathcal{M}_i \vert \mathcal{D})$. Without having looked at whether $\mathbf{x}$ is a test-data point or validation-data point, I cannot be sure of this currently. Perhaps I will edit when I get time to revisit the chapter. $\endgroup$
    – microhaus
    Commented Jul 6, 2021 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.