2
$\begingroup$

Currently reading from Christopher Bishop's Pattern Recognition and Machine Learning book about parameter distribution under a bayesian linear regression.

On page 153, the author deduces that the posterior distribution over weights for a model of the form $y(\bf x, \bf w) + \varepsilon$ after $N$ observations with $y({\bf x}, {\bf w}) = {\bf w}^T{\bf \phi(x)}$, $p(\varepsilon) = \mathcal{N}(0, \beta)$ and with a conjugate prior of the form

$$ p({\bf w}) = \mathcal{N}({\bf w} | {\bf m}_0, {\bf S}_0) $$

is

$$ \begin{align} {\bf m}_N &= {\bf S}_N({\bf S}_0^{-1}{\bf m}_0 + \beta{\bf \Phi}^T{\bf t}) \\ {\bf S}_N^{-1} &= {\bf S}_0^{-1} + \beta{\bf \Phi}^T{\bf \Phi} \end{align} $$

Then, he continues by saying that

If we consider an infinitely broad prior ${\bf S}_0 = \alpha^{-1}{\bf I}$ with $\alpha \to 0$, the mean ${\bf m}_N$ of the posterior distribution reduces to the maximum likelihood value ${\bf w}_{\text{ML}}$

Where ${\bf w}_{\text{ML}} =({\bf \Phi}^T{\bf \Phi})^{-1}{\bf \Phi}{\bf t}$.

In doing so, I arrive at $$ {\bf m}_N = {\bf m}_0 + \alpha^{-1}\beta{\bf \Phi}^T{\bf t} + \alpha\beta^{-1}({\bf \Phi}^T{\bf \Phi})^{-1}{\bf m}_0 + ({\bf \Phi}^T{\bf \Phi})^{-1}{\bf \Phi}{\bf t} $$

For which the only way to get rid of ${\bf m}_0$ if it is a zero vector. Thus, we should consider a zero mean isotropic gaussian as a prior. In doing so I arrive at

$$ {\bf m}_N = \alpha^{-1}\beta{\bf \Phi}^T{\bf t} + ({\bf \Phi}^T{\bf \Phi})^{-1}{\bf \Phi}{\bf t} $$

Finally, if I take the limit $\alpha \to 0$ in ${\bf m}_0$, then $\lim_{\alpha \to 0^+} 1/\alpha = \infty $ and $\lim_{\alpha \to 0^-} 1/\alpha = -\infty $, which does not converge and cannot reduced ${\bf m}_N$ to ${\bf w}_{\text{ML}}$. Since we want an infinitely broad prior, ${\bf S}_0$ has to be ${\bf S}_0 = \alpha^{-1}{\bf I}$ with $\alpha \to 0$.

What's the argument I need to conclude that ${\bf m}_N$ does indeed converge to ${\bf w}_{\text{ML}}$?

$\endgroup$
1
  • $\begingroup$ I'm reading this part and completely dumbfounded as to how the author got these calculations for $$m_N$$ and $$S_N^{-1}$$. Care to explain how he got there? $\endgroup$
    – doctopus
    Commented Oct 22, 2019 at 22:44

1 Answer 1

1
$\begingroup$

From the formulas for $\mathbf{m_N}$ and $\mathbf{S_N}^{-1}$, I arrive at: $$\mathbf{S_N^{-1} m_N=\alpha I\ m_0}+\beta\mathbf{\Phi^Tt}\rightarrow (\alpha\bf I + \beta{\bf \Phi}^T{\bf \Phi})\mathbf{m_N=\alpha\bf I\ m_0}+\beta\mathbf{\Phi^Tt}$$ Taking $\alpha\rightarrow 0$ in both sides of the equation leaves us with the following: $$\beta\ {\bf \Phi}^T{\bf \Phi}\ \mathbf{m_N}=\beta\ \mathbf{\Phi^Tt}\rightarrow \bf m_N=({\bf \Phi^T\Phi})^{-1}\bf \Phi^T\bf t$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.