Pattern Recognition and Machine Learning (Bishop) - How is this log-evidence function maximized with respect to $\alpha$?

Question

In the book Pattern Recognition and Machine Learning, the author writes the log-evidence function (equation 3.86 in page 167):

ln $p(\textbf{t}| \alpha, \beta) = \frac{M}{2}$ ln $\alpha$ + $\frac{N}{2}$ ln $\beta$ - $E (\textbf{m}_N)$ - $\frac{1}{2}$ ln|$\textbf{A}$| - $\frac{N}{2}$ ln($2\pi$)

where $E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi \, \textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2 $ (equation 3.82)

Then, he differentiates the log-evidence with respect to $\alpha$ and sets to zero (to maximize), getting:

$0 = \frac{M}{2\alpha} - \frac{1}{2} || \textbf{m}_N ||^2 - \frac{1}{2} \sum_i{\frac{1}{\lambda_i + \alpha}}$ (equation 3.89)

The last term is $\frac{1}{2} \frac{d}{d \alpha}$ ln |$\textbf{A}$|, it is not important in this question and I have already checked it.

The thing is that, as far as I understand, the log-evidence has been differentiated as if $\textbf{m}_N$ was constant with respect to $\alpha$, which is not the case, because in page 153 we can see its definition:

$\textbf{m}_N = \beta \;\textbf{S}_N \; \Phi^T \textbf{t}$ (equation 3.53)

where $\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi \Phi^T$ (equation 3.54)

I would appreciate any help to understand the derivation of 3.89 by differentiation of 3.86 with respect to $\alpha$.

Answer 1

Continuing with your notation: $$E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi\textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2$$ $$ = \frac{\beta}{2}(\textbf{t} - \Phi\textbf{m}_N)^T(\textbf{t} - \Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ $$=\frac{\beta}{2}(\textbf{t}^T\textbf{t} -2\textbf{t}^T\Phi\textbf{m}_N+\textbf{m}_N^T\Phi^T\Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ So $$\frac{d}{d\alpha}E(\textbf{m}_N)=\beta(\textbf{m}_N^T\Phi^T\Phi-\textbf{t}^T\Phi)\frac{d}{d\alpha}\textbf{m}_N+\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\alpha\textbf{m}_N^T \frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\{\textbf{m}_N^T(\alpha \textbf{I}+\beta\Phi^T\Phi)-\beta\textbf{t}^T\Phi\}\frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N $$where the term in curly braces vanishes by eqs. 3.53 and 3.54($\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi^T\Phi $) above: $$\textbf{m}_N^T\textbf{S}_N^{-1}=\beta\textbf{t}^T\Phi $$

So it is not obvious that the additional $\alpha$ dependence of $E (\textbf{m}_N)$ that you point out has vanishing derivative, but there it is, it does. I too was puzzled when I saw no mention of it in the text, or in the solution posted for exercise 3.20 asking to deriver the result, which is therefore rather incomplete. A similar thing happens when maximizing the evidence wrt to $\beta$.

Answer 2

I had the same question...

Let $E(\mathbf{m}, \alpha, \beta) = \frac{\beta}{2} || y - X \mathbf{m} ||^2 + \frac{\alpha}{2} ||\mathbf{m}||^2$

$\mathbf{m}_n(\alpha, \beta)$ is chosen to minimize $E$, so we have

$$\frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) = 0$$

Finally the derivatives of interest

\begin{align} \frac{\partial}{\partial \alpha}E(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) &= \frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \frac{\partial \mathbf{m}_n}{\partial \alpha}(\alpha, \beta) + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= 0 + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= \frac{1}{2} ||\mathbf{m}_n(\alpha, \beta)||^2 \end{align}

Same for $\beta$.