1
$\begingroup$

In Pattern Recognition and Machine Learning Chapter 6.4.6 the author applies the Laplace approximation to Gaussian process regression and in doing so has to maximise the log likelihood $ln p(t_N | \theta)$ in order to obtain the parameters $\theta$ of the covariance function.

I am trying to derive expression 6.91 for the derivative of the terms involving $C_N$ by following along with the tutors solution manual for the book. Following the solution manual if you gather the terms that involve $C_N$ you get:

$$-\frac{1}{2}a_N^{*T}C_N^{-1}a_N^*-\frac{1}{2}ln |C_N W_N +I|$$

as given in the solution manual. The author then states to apply C.21 and C.22 to the above to get the derivative wrt $\theta_j$:

$$\frac{\partial}{\partial x}A^{-1}=-A^{-1}\frac{\partial A}{\partial x}A^{-1}$$

and C.22:

$$\frac{\partial }{\partial x} ln |A| = Tr(A^{-1} \frac{\partial A}{\partial x})$$

If you apply these you get:

$$\frac{1}{2}a_N^{*T}C_N^{-1}\frac{\partial C_N}{\partial \theta_j}C_N^{-1}a_N^*-\frac{1}{2}Tr((C_N W_N + I)^{-1}\frac{\partial C_N}{\partial \theta_j}W_N)$$

Since $W_N$ is constant wrt $\theta_j$. Whereas the formula in the book has the following for the trace term:

$$-\frac{1}{2}Tr((C_N W_N + I)^{-1}W_N\frac{\partial C_N}{\partial \theta_j})$$

i.e. the order of the last two matrices is swapped.

I'm fairly sure this is a typo since $$Tr(ABC) \neq Tr(ACB)$$

I couldn't find this listed in any errata online. I found an unofficial solutions manual which says it thinks 6.91 is incorrect but it wasn't stated what they thought the error was so I wanted to check that my reasoning is sound and that this is indeed an error.

Note $C_N$ is symmetric.

EDIT: There's a similar typo listed in this unofficial errata which is one of the top search results on Google for 'PRML errata', but it concerns equation 6.93 and there is nothing listed in there or the official errata about 6.91.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't think the appendix is consistent. It depends on which layout notation you use while calculating the derivatives. For example, C.18 & C.20 uses numerator notation, but C.19 uses denominator notation. You can't combine formulas with different layouts safely. But, I think the book usually follows the numerator notation without explicitly stating it.

The derivative for $\ln |A|$ is notation agnostic and is correct. But, C.22 is in numerator notation, and the rest should be in the same notation for consistency. Therefore, following numerator notation, your formula is the correct one.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think the book actually follows denominator layout mostly. $\endgroup$
    – tail_recursion
    Mar 25, 2022 at 2:06
  • $\begingroup$ Yet, it should have chosen and followed a consistent layout; or mentioned it whenever necessary. $\endgroup$
    – gunes
    Mar 26, 2022 at 17:02
  • 1
    $\begingroup$ Yeah. I think if you have never seen matrix calculus before it would be worth looking at other resources before trying to read the book. Out of all the math topics that people might not have seen before in PRML, matrix calculus is probably the most important IMO yet how it is introduced and explained overall is pretty poor if you have never seen it before IMO, along with the inconsistency with the layout. It is probably one of the weakest aspects of the book and I say that even though it is easily one of my favourite books. $\endgroup$
    – tail_recursion
    Mar 27, 2022 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.