4
$\begingroup$

In the book Pattern Recognition and Machine Learning, the author writes the log-evidence function (equation 3.86 in page 167):

ln $p(\textbf{t}| \alpha, \beta) = \frac{M}{2}$ ln $\alpha$ + $\frac{N}{2}$ ln $\beta$ - $E (\textbf{m}_N)$ - $\frac{1}{2}$ ln|$\textbf{A}$| - $\frac{N}{2}$ ln($2\pi$)

where $E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi \, \textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2 $ (equation 3.82)

Then, he differentiates the log-evidence with respect to $\alpha$ and sets to zero (to maximize), getting:

$0 = \frac{M}{2\alpha} - \frac{1}{2} || \textbf{m}_N ||^2 - \frac{1}{2} \sum_i{\frac{1}{\lambda_i + \alpha}}$ (equation 3.89)

The last term is $\frac{1}{2} \frac{d}{d \alpha}$ ln |$\textbf{A}$|, it is not important in this question and I have already checked it.

The thing is that, as far as I understand, the log-evidence has been differentiated as if $\textbf{m}_N$ was constant with respect to $\alpha$, which is not the case, because in page 153 we can see its definition:

$\textbf{m}_N = \beta \;\textbf{S}_N \; \Phi^T \textbf{t}$ (equation 3.53)

where $\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi \Phi^T$ (equation 3.54)

I would appreciate any help to understand the derivation of 3.89 by differentiation of 3.86 with respect to $\alpha$.

$\endgroup$

2 Answers 2

4
$\begingroup$

Continuing with your notation: $$E (\textbf{m}_N) = \frac{\beta}{2} || \textbf{t} - \Phi\textbf{m}_N ||^2 + \frac{\alpha}{2} || \textbf{m}_N ||^2$$ $$ = \frac{\beta}{2}(\textbf{t} - \Phi\textbf{m}_N)^T(\textbf{t} - \Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ $$=\frac{\beta}{2}(\textbf{t}^T\textbf{t} -2\textbf{t}^T\Phi\textbf{m}_N+\textbf{m}_N^T\Phi^T\Phi\textbf{m}_N)+\frac{\alpha}{2}\textbf{m}_N^T\textbf{m}_N$$ So $$\frac{d}{d\alpha}E(\textbf{m}_N)=\beta(\textbf{m}_N^T\Phi^T\Phi-\textbf{t}^T\Phi)\frac{d}{d\alpha}\textbf{m}_N+\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\alpha\textbf{m}_N^T \frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N+\{\textbf{m}_N^T(\alpha \textbf{I}+\beta\Phi^T\Phi)-\beta\textbf{t}^T\Phi\}\frac{d}{d\alpha}\textbf{m}_N$$ $$=\frac{1}{2}\textbf{m}_N^T\textbf{m}_N $$where the term in curly braces vanishes by eqs. 3.53 and 3.54($\textbf{S}_N ^{-1} = \alpha \textbf{I} + \beta \; \Phi^T\Phi $) above: $$\textbf{m}_N^T\textbf{S}_N^{-1}=\beta\textbf{t}^T\Phi $$

So it is not obvious that the additional $\alpha$ dependence of $E (\textbf{m}_N)$ that you point out has vanishing derivative, but there it is, it does. I too was puzzled when I saw no mention of it in the text, or in the solution posted for exercise 3.20 asking to deriver the result, which is therefore rather incomplete. A similar thing happens when maximizing the evidence wrt to $\beta$.

$\endgroup$
1
  • $\begingroup$ This thing bothered me for a long time. Very simple answer, thanks! :D $\endgroup$
    – Javi
    Apr 26, 2019 at 5:27
0
$\begingroup$

I had the same question...

Let $E(\mathbf{m}, \alpha, \beta) = \frac{\beta}{2} || y - X \mathbf{m} ||^2 + \frac{\alpha}{2} ||\mathbf{m}||^2$

$\mathbf{m}_n(\alpha, \beta)$ is chosen to minimize $E$, so we have

$$\frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) = 0$$

Finally the derivatives of interest

\begin{align} \frac{\partial}{\partial \alpha}E(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) &= \frac{\partial E}{\partial \mathbf{m}}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \frac{\partial \mathbf{m}_n}{\partial \alpha}(\alpha, \beta) + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= 0 + \frac{\partial E}{\partial \alpha}(\mathbf{m}_n(\alpha, \beta), \alpha, \beta) \\ &= \frac{1}{2} ||\mathbf{m}_n(\alpha, \beta)||^2 \end{align}

Same for $\beta$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.