I am studying (myself, not in class) the book of Rogers & Girolami, A First Course in Machine Learning.

In working through a logistic classifier, I found the equation $$ p(t_{new} = 1| \mathbf{x}_{new}, \mathbf{X}, \mathbf{t}) = E_{p(\mathbf{w}|\mathbf{X,t})} \left\{ \frac{1}{1 + \exp(-\mathbf{w}^T \mathbf{x}_{new})} \right\} $$

So is this the expectation with respect to the posterior... of the probability of a new data point being 1?

I do not recognize this expression. Does it have a name?

My question: can someone describe this in a more generic form? For example is it equivalent $$ \int p(D=\text{const}|M)\, p(M|D)\, dM $$ where $D$ is "data" and $M$ is "model" ? If so, then the expression $\int p(D=\text{const}|M) dM $ does not integrate to 1 I think, it feels wrong.

  • $\begingroup$ The link goes to the wrong book on Amazon. $\endgroup$ – Vladislavs Dovgalecs Apr 9 '18 at 6:51
  • 1
    $\begingroup$ I removed the link. That is weird, I did not add the link. Maybe it is added auotmatically? $\endgroup$ – matchingmoments Apr 9 '18 at 19:06

In a Bayesian perspective, the predictive distribution is constructed by incorporating a new value of the observable $t_\text{new}$ as a part of the unknowns, hence aggregating it with the parameter $w$. The predictive distribution is thus derived by marginalising over the parameter: $$p(t_\text{new}|{\cal D})=\int p(t_\text{new},w|{\cal D})\, \mathrm{d} w \propto \int p(t_\text{new}|w)p({\cal D}|w) p(w)\, \mathrm{d} w$$ Hence, in the logistic case $$pr(t_\text{new}=1|{\cal D})=\int pr(t_\text{new}=1,w|{\cal D})\, \mathrm{d} w \propto \int pr(t_\text{new}=1|w)p({\cal D}|w) p(w)\, \mathrm{d} w$$or $$pr(t_\text{new}=1|{\cal D})=\mathbb{E}^\pi[pr(t_\text{new}=1|w)|{\cal D}]$$

  • 1
    $\begingroup$ There seems some typos, it should be : ∫p(tnew,w|D)dw∝∫p(tnew|w)p(w|D)dw also, ∫ pr(tnew=1,w|D)dw∝∫pr(tnew=1|w)p(w|D)dw $\endgroup$ – lynnjohn Apr 9 '18 at 6:41
  • $\begingroup$ @lynnjohn Right, the prior is not necessarily a flat prior. $\endgroup$ – Xi'an Apr 9 '18 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.