The confusing derivation in the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy

Question

In the section 15.5 of the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy, it discusses the Gaussian Process Latent Variable Model. The log-likelihood objective function is given By

$$l = -\frac{D}{2}\ln|K| - \frac{1}{2}\text{tr}(K^{-1}YY^{T})\tag 1$$

Where $K = ZZ^T + \beta^{-1}I$

and the gradient with regard to Z is given by:

$$\frac{\partial l}{\partial \mathbf{Z}_{ij}} = \frac{\partial l}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{Z}_{ij}}\tag 2$$ and

$$\frac{\partial l}{\partial K} = K^{-1}YY^TK^{-1} - DK^{-1}\tag 3$$

(I think the author omits the '$ \frac{1}{2}$' here).

Anyway, I can get the equation (3) by the rules in MatrixCookbook. The author then says we can have

$$\frac{\partial K}{\partial Z} = Z$$

(I think the author omits 2 here)

Finally, we get

$$\frac{\partial l}{\partial Z} = K^{-1}YY^TK^{-1}Z - DK^{-1}Z\tag 4$$

The result matches the result in Lawrence 2005. And there is a similar derivation in this site, see the answer. It seems that the chain rule($$\frac{\partial l}{\partial \mathbf{Z}} = \frac{\partial l}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{Z}}\tag 5$$) works here.

But as I know, only $$\frac{\partial \text{Tr}[K]}{\partial Z}= 2Z\tag 6$$ and $$\frac{\partial l}{\partial \mathbf{Z}_{ij}} = \text{Tr}\left[\left(\frac{\partial l}{\partial \mathbf{K}}\right)^T\frac{\partial \mathbf{K}}{\partial \mathbf{Z}_{ij}}\right]\tag 7$$. I assume the author omits the'Tr', but How can I get (4) by using (6) and (7).And are there any connections between (5) and (7)? As far as I know, the equation (5) is invalid for matrix-multiplying-shape match.

Answer 1

$ \def\d{D} \def\l{\ell} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\K{K^{-1}} $First, introduce the extremely useful Frobenius product $(:)\:$ which has the following properties $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \frob{A}^2 \qquad \{ {\rm Frobenius\;norm} \}\\ A:B &= B:A \;=\; A^T:B^T \\ {CA}:B &= C:BA^T \;=\; A:C^TB \\ }$$ So part of the answer to your question is that the Frobenius product gets rid of the trace function, so equation $(5)$ should be using the Frobenius product and not a standard matrix product. Also note that $K \LR{{\rm as\;well\;as\;}\grad{\l}{K}}$ is a symmetric matrix, which simplifies the Frobenius product formulas. Finally, $\grad KZ$ is a fourth-order tensor, which is a PITA to work with using standard matrix-vector notation.

So, instead of the chain rule (which requires the tensor-valued gradient), use the matrix-valued gradient $\grad{\l}{K}$ to write the differential of $\l,\,$ then change the independent variable from $K\to Z\,$ and formulate the desired gradient wrt $Z$ $$\eqalign{ d\l &= \frac12\LR{\K YY^T\K-\d\K}:dK \\ &= \frac12\LR{\K YY^T\K-\d\K}:\LR{Z\:dZ^T + dZ\:Z^T} \\ &= \LR{\K YY^T\K-\d\K}:\LR{dZ\:Z^T} \\ &= \LR{\K YY^T\K Z-\d\K Z}:dZ \\ \grad{\l}{Z} &= {\K YY^T\K Z-\d\K Z} \\ }$$