I am attempting to implement the Nonlinear Gaussian Process Latent Variable Model, as per Lawrence 2005 and have the gradient with respect to the kernel as follows(Eq 10 in paper):

$\frac{\partial L}{\partial \mathbf{K}} = K^{-1}YY^{T}K^{-1} -DK^{-1}$

where $K$ is an $n \times n$ covariance matrix(built from an initial latent space representation of $Y$) and $n$ is the number of data points and $Y$ is an $n \times d$ matrix of $d$ dimensional data in the observation space.

I am looking to compute the following gradients:

$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{x}_{n}}$


$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \theta}$

where $\mathbf{x}_{n}$ is the vector of the n'th row of $X$(the initial $n \times q$ latent space - obtained with PCA) and $\theta$ is an arbitrary hyperparameter.

The gradient $\frac{\partial L}{\partial \mathbf{K}}$ is an $n \times n$ matrix, as is the gradient $\frac{\partial \mathbf{K}}{\partial \theta}$.

Two things are not clear at this stage:

1) Taking the gradient w.r.t. a $q$ dimensional feature vector of the covariance matrix yields an $n \times n \times q$ tensor, by evaluating the partial derivative of the covariance function for each pair of points w.r.t the first $q$ dimensional vector argument($a$ for instance). How can one combine this tensor of gradients with $\frac{\partial L}{\partial \mathbf{K}}$ to form a $n \times q$ update matrix for the latent space points?

2) When taking the gradient w.r.t. some arbitrary hyperparameter $\theta$ one obtains an $n \times n$ matrix by evaluating the partial of the covariance function for each pair of data points, in much the same way as the covariance matrix is generated. To produce a scalar update gradient for the hyperparameter, does one just aggregate the gradients of all the derivative covariance matrix values?


After further research, I have found an SGD algorithm for online GP-LVM training on slide 62 found here. So, if we have a latent space vector $\mathbf{x}_{n}$ and the gradient $\frac{\partial L}{\partial \mathbf{x}_{n}}$ is it valid to update all latent space points with this gradient for the current epoch(ignoring the R neighbours, for now)? The gradient would be an $n \times q$ matrix.

  • $\begingroup$ Hi! Could you please add the full references of the papers you cite in case your links die in the future? Thanks! $\endgroup$ – Antoine Jul 2 '20 at 19:01

Although my answer is late, I thought it may help other people.

Let's first recall the log-likelihood objective function in question (Eq.6, Lawrence2005):

$$L = -\frac{DN}{2}ln 2\pi - \frac{D}{2}ln|K| - \frac{1}{2}tr(K^{-1}YY^{T}),$$

where $$ K = XX^T + \beta^{-1}I$$

The gradient of L w.r.t X is given as follow:

$$\frac{\partial L}{\partial X} = \frac{\partial}{\partial X}(-\frac{DN}{2}ln 2\pi) + \frac{\partial}{\partial X}(- \frac{D}{2}ln|K|) + \frac{\partial}{\partial X}(- \frac{1}{2}tr(K^{-1}YY^{T}))$$

The derivative of the first term $$\frac{\partial}{\partial X}(-\frac{DN}{2}ln 2\pi) = 0$$ as it's a constant.

To obtain the derivative of the second term we use the chain rule, i.e. $$ [f(g(x))]' = f'(g(x))g'(x)$$

So : $$ \frac{\partial}{\partial X}(- \frac{D}{2}ln|K|) = \frac{-D}{2}(\frac{\partial ln|K|}{\partial K})(\frac{\partial K}{\partial X}) = \frac{-D}{2}(\frac{1}{K})(2X) = -DK^{-1}X$$

The same for the last term: $$ \frac{\partial}{\partial X}(- \frac{1}{2}tr(K^{-1}YY^{T})) = \frac{-1}{2}\frac{\partial tr(K^{-1}YY^T)}{\partial K}\frac{\partial K}{\partial X}$$

We have (See MatrixCookbook) $$ \frac{\partial}{\partial Z}tr(AZ^{-1}B) = -(Z^{-1}BAZ^{-1})^T$$

By putting $$A=I, Z=K, B=YY^T$$ we can obtain

$$ \frac{-1}{2}\frac{\partial tr(K^{-1}YY^T)}{\partial K}\frac{\partial K}{\partial X} = \frac{-1}{2}(-1)((K^{-1})^TI^T(YY^T)^T(K^{-1})^T)(2X) = K^{-1}YY^TK^{-1}X$$

We finally obtain $$ \frac{\partial L}{\partial X} = K^{-1}YY^TK^{-1}X - DK^{-1}X $$

For the $\frac{\partial L}{\partial K}$, we need to calculate the derivative of L w.r.t K only, so we just drop the $\frac{\partial K}{\partial X}$ term from the above equation:

$$ \frac{\partial L}{\partial K} = K^{-1}YY^TK^{-1} - DK^{-1} $$

The derivative of L w.r.t $\theta$, i.e. $\frac{\partial L}{\partial \theta}$, is zero simply because $\theta$ is not a variable of L.


I am facing the same today. However I should say the the ultimate gradient is computed by taking the trace, so at the end it is a scalar. Denoting $x_i$ as the i-th component of vector $x$, and L the log-likelihood:

$\frac{\partial L}{\partial x_{i}}= \text{Tr}[\frac{\partial L}{\partial K}\frac{\partial K}{\partial x_{i,j}}]$

So I think we should compute the second term and compute the trace of that matrix product, which can be done by computing the product of the rows of $\frac{\partial L}{\partial K}$ with the columns of $\frac{\partial K}{\partial x_{i,j}}$ and summing up this dots products. I think that those slides do not have the answer we are looking for. Note that this answer answers both of your questions.

  • $\begingroup$ This doesn't seem to answer the question. $\endgroup$ – Michael R. Chernick Oct 23 '19 at 16:28
  • $\begingroup$ Yes, just consider his 2) question. The way you reduce the matrix to scalar is because you take the trace of that product. $\endgroup$ – jdeJuan Oct 23 '19 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.