In the section 15.5 of the book 'Machine Learning: A Probabilistic Perspective' by Kevin P. Murphy, it discusses the Gaussian Process Latent Variable Model. The log-likelihood objective function is given By

$$l = -\frac{D}{2}\ln|K| - \frac{1}{2}\text{tr}(K^{-1}YY^{T})\tag 1$$

Where $K = ZZ^T + \beta^{-1}I$

and the gradient with regard to Z is given by:

$$\frac{\partial l}{\partial \mathbf{Z}_{ij}} = \frac{\partial l}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{Z}_{ij}}\tag 2$$ and

$$\frac{\partial l}{\partial K} = K^{-1}YY^TK^{-1} - DK^{-1}\tag 3$$

(I think the author omits the '$ \frac{1}{2}$' here).

Anyway, I can get the equation (3) by the rules in MatrixCookbook. The author then says we can have

$$\frac{\partial K}{\partial Z} = Z$$

(I think the author omits 2 here)

Finally, we get

$$\frac{\partial l}{\partial Z} = K^{-1}YY^TK^{-1}Z - DK^{-1}Z\tag 4$$

The result matches the result in Lawrence 2005. And there is a similar derivation in this site, see the answer. It seems that the chain rule($$\frac{\partial l}{\partial \mathbf{Z}} = \frac{\partial l}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{Z}}\tag 5$$) works here.

But as I know, only $$\frac{\partial \text{Tr}[K]}{\partial Z}= 2Z\tag 6$$ and $$\frac{\partial l}{\partial \mathbf{Z}_{ij}} = \text{Tr}\left[\left(\frac{\partial l}{\partial \mathbf{K}}\right)^T\frac{\partial \mathbf{K}}{\partial \mathbf{Z}_{ij}}\right]\tag 7$$. I assume the author omits the'Tr', but How can I get (4) by using (6) and (7).And are there any connections between (5) and (7)? As far as I know, the equation (5) is invalid for matrix-multiplying-shape match.


1 Answer 1


$ \def\d{D} \def\l{\ell} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\trace#1{\op{Tr}\LR{#1}} \def\frob#1{\left\| #1 \right\|_F} \def\qiq{\quad\implies\quad} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\K{K^{-1}} $First, introduce the extremely useful Frobenius product $(:)\:$ which has the following properties $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{A^TB} \\ A:A &= \frob{A}^2 \qquad \{ {\rm Frobenius\;norm} \}\\ A:B &= B:A \;=\; A^T:B^T \\ {CA}:B &= C:BA^T \;=\; A:C^TB \\ }$$ So part of the answer to your question is that the Frobenius product gets rid of the trace function, so equation $(5)$ should be using the Frobenius product and not a standard matrix product. Also note that $K \LR{{\rm as\;well\;as\;}\grad{\l}{K}}$ is a symmetric matrix, which simplifies the Frobenius product formulas. Finally, $\grad KZ$ is a fourth-order tensor, which is a PITA to work with using standard matrix-vector notation.

So, instead of the chain rule (which requires the tensor-valued gradient), use the matrix-valued gradient $\grad{\l}{K}$ to write the differential of $\l,\,$ then change the independent variable from $K\to Z\,$ and formulate the desired gradient wrt $Z$ $$\eqalign{ d\l &= \frac12\LR{\K YY^T\K-\d\K}:dK \\ &= \frac12\LR{\K YY^T\K-\d\K}:\LR{Z\:dZ^T + dZ\:Z^T} \\ &= \LR{\K YY^T\K-\d\K}:\LR{dZ\:Z^T} \\ &= \LR{\K YY^T\K Z-\d\K Z}:dZ \\ \grad{\l}{Z} &= {\K YY^T\K Z-\d\K Z} \\ }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.