Pattern Recognition and Machine Learning (Bishop) - How is this log-evidence function maximized with respect to $\alpha$? In the book Pattern Recognition and Machine Learning, the author writes the log-evidence function (equation 3.86 in page 167):
ln $p(\textbf{t}| \alpha, \beta) = \frac{M}{2}$ ln $\alpha$ + $\frac{N}{2}$ ln $\beta$ - $E (\textbf{m}_N)$ - $\frac{1}{2}$ ln|$\textbf{A}$| - $\frac{N}{2}$ ln($2\pi$)
where $E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi \, \textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2 $ (equation 3.82)
Then, he differentiates the log-evidence with respect to $\alpha$ and sets to zero (to maximize), getting:
$0 = \frac{M}{2\alpha} - \frac{1}{2} || \textbf{m}_N ||^2 - \frac{1}{2} \sum_i{\frac{1}{\lambda_i + \alpha}}$ (equation 3.89)
The last term is $\frac{1}{2} \frac{d}{d \alpha}$ ln |$\textbf{A}$|, it is not important in this question and I have already checked it.
The thing is that, as far as I understand, the log-evidence has been differentiated as if $\textbf{m}_N$ was constant with respect to $\alpha$, which is not the case, because in page 153 we can see its definition:
$\textbf{m}_N = \beta \;\textbf{S}_N \; \Phi^T \textbf{t}$ (equation 3.53)
where $\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi \Phi^T$ (equation 3.54)
I would appreciate any help to understand the derivation of 3.89 by differentiation of 3.86 with respect to $\alpha$.
 A: Continuing with your notation:
$$E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi\textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2$$ $$ = \frac{\beta}{2}(\textbf{t} - \Phi\textbf{m}_N)^T(\textbf{t} - \Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$
$$=\frac{\beta}{2}(\textbf{t}^T\textbf{t} -2\textbf{t}^T\Phi\textbf{m}_N+\textbf{m}_N^T\Phi^T\Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$
So
$$\frac{d}{d\alpha}E(\textbf{m}_N)=\beta(\textbf{m}_N^T\Phi^T\Phi-\textbf{t}^T\Phi)\frac{d}{d\alpha}\textbf{m}_N+\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\alpha\textbf{m}_N^T \frac{d}{d\alpha}\textbf{m}_N$$
$$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\{\textbf{m}_N^T(\alpha \textbf{I}+\beta\Phi^T\Phi)-\beta\textbf{t}^T\Phi\}\frac{d}{d\alpha}\textbf{m}_N$$
$$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N $$where the term in curly braces vanishes by eqs. 3.53 and 3.54($\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi^T\Phi $) above:
$$\textbf{m}_N^T\textbf{S}_N^{-1}=\beta\textbf{t}^T\Phi $$
So it is not obvious that the additional $\alpha$ dependence of $E (\textbf{m}_N)$ that you point out has vanishing derivative, but there it is, it does. I too was puzzled when I saw no mention of it in the text, or in the solution posted for exercise 3.20 asking to deriver the result, which is therefore rather incomplete. A similar thing happens when maximizing the evidence wrt to $\beta$.
A: I had the same question...
Let $E(\mathbf{m}, \alpha, \beta) = \frac{\beta}{2} || y - X \mathbf{m} ||^2 + \frac{\alpha}{2} ||\mathbf{m}||^2$
$\mathbf{m}_n(\alpha, \beta)$ is chosen to minimize $E$, so we have
$$\frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) = 0$$
Finally the derivatives of interest
\begin{align}
\frac{\partial}{\partial \alpha}E(\mathbf{m}_n(\alpha, \beta), \alpha, \beta)
&= \frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \frac{\partial \mathbf{m}_n}{\partial \alpha}(\alpha, \beta)
+ \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\
&= 0
+ \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\
&=  \frac{1}{2} ||\mathbf{m}_n(\alpha, \beta)||^2
\end{align}
Same for $\beta$.
