Pattern Recognition and Machine Learning (Bishop) - How is this log-evidence function maximized with respect to $\alpha$?

In the book Pattern Recognition and Machine Learning, the author writes the log-evidence function (equation 3.86 in page 167):

ln $$p(\textbf{t}| \alpha, \beta) = \frac{M}{2}$$ ln $$\alpha$$ + $$\frac{N}{2}$$ ln $$\beta$$ - $$E (\textbf{m}_N)$$ - $$\frac{1}{2}$$ ln|$$\textbf{A}$$| - $$\frac{N}{2}$$ ln($$2\pi$$)

where $$E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi \, \textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2$$ (equation 3.82)

Then, he differentiates the log-evidence with respect to $$\alpha$$ and sets to zero (to maximize), getting:

$$0 = \frac{M}{2\alpha} - \frac{1}{2} || \textbf{m}_N ||^2 - \frac{1}{2} \sum_i{\frac{1}{\lambda_i + \alpha}}$$ (equation 3.89)

The last term is $$\frac{1}{2} \frac{d}{d \alpha}$$ ln |$$\textbf{A}$$|, it is not important in this question and I have already checked it.

The thing is that, as far as I understand, the log-evidence has been differentiated as if $$\textbf{m}_N$$ was constant with respect to $$\alpha$$, which is not the case, because in page 153 we can see its definition:

$$\textbf{m}_N = \beta \;\textbf{S}_N \; \Phi^T \textbf{t}$$ (equation 3.53)

where $$\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi \Phi^T$$ (equation 3.54)

I would appreciate any help to understand the derivation of 3.89 by differentiation of 3.86 with respect to $$\alpha$$.

Continuing with your notation: $$E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi\textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2$$ $$= \frac{\beta}{2}(\textbf{t} - \Phi\textbf{m}_N)^T(\textbf{t} - \Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ $$=\frac{\beta}{2}(\textbf{t}^T\textbf{t} -2\textbf{t}^T\Phi\textbf{m}_N+\textbf{m}_N^T\Phi^T\Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ So $$\frac{d}{d\alpha}E(\textbf{m}_N)=\beta(\textbf{m}_N^T\Phi^T\Phi-\textbf{t}^T\Phi)\frac{d}{d\alpha}\textbf{m}_N+\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\alpha\textbf{m}_N^T \frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\{\textbf{m}_N^T(\alpha \textbf{I}+\beta\Phi^T\Phi)-\beta\textbf{t}^T\Phi\}\frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N$$where the term in curly braces vanishes by eqs. 3.53 and 3.54($$\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi^T\Phi$$) above: $$\textbf{m}_N^T\textbf{S}_N^{-1}=\beta\textbf{t}^T\Phi$$

So it is not obvious that the additional $$\alpha$$ dependence of $$E (\textbf{m}_N)$$ that you point out has vanishing derivative, but there it is, it does. I too was puzzled when I saw no mention of it in the text, or in the solution posted for exercise 3.20 asking to deriver the result, which is therefore rather incomplete. A similar thing happens when maximizing the evidence wrt to $$\beta$$.

• This thing bothered me for a long time. Very simple answer, thanks! :D
– Javi
Apr 26, 2019 at 5:27

Let $$E(\mathbf{m}, \alpha, \beta) = \frac{\beta}{2} || y - X \mathbf{m} ||^2 + \frac{\alpha}{2} ||\mathbf{m}||^2$$
$$\mathbf{m}_n(\alpha, \beta)$$ is chosen to minimize $$E$$, so we have
$$\frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) = 0$$
\begin{align} \frac{\partial}{\partial \alpha}E(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) &= \frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \frac{\partial \mathbf{m}_n}{\partial \alpha}(\alpha, \beta) + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= 0 + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= \frac{1}{2} ||\mathbf{m}_n(\alpha, \beta)||^2 \end{align}
Same for $$\beta$$.