Gaussian Process Latent Variable Model Optimisation

Question

I am attempting to implement the Nonlinear Gaussian Process Latent Variable Model, as per Lawrence 2005 and have the gradient with respect to the kernel as follows(Eq 10 in paper):

$\frac{\partial L}{\partial \mathbf{K}} = K^{-1}YY^{T}K^{-1} -DK^{-1}$

where $K$ is an $n \times n$ covariance matrix(built from an initial latent space representation of $Y$) and $n$ is the number of data points and $Y$ is an $n \times d$ matrix of $d$ dimensional data in the observation space.

I am looking to compute the following gradients:

$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{x}_{n}}$

and

$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \theta}$

where $\mathbf{x}_{n}$ is the vector of the n'th row of $X$(the initial $n \times q$ latent space - obtained with PCA) and $\theta$ is an arbitrary hyperparameter.

The gradient $\frac{\partial L}{\partial \mathbf{K}}$ is an $n \times n$ matrix, as is the gradient $\frac{\partial \mathbf{K}}{\partial \theta}$.

Two things are not clear at this stage:

1) Taking the gradient w.r.t. a $q$ dimensional feature vector of the covariance matrix yields an $n \times n \times q$ tensor, by evaluating the partial derivative of the covariance function for each pair of points w.r.t the first $q$ dimensional vector argument($a$ for instance). How can one combine this tensor of gradients with $\frac{\partial L}{\partial \mathbf{K}}$ to form a $n \times q$ update matrix for the latent space points?

2) When taking the gradient w.r.t. some arbitrary hyperparameter $\theta$ one obtains an $n \times n$ matrix by evaluating the partial of the covariance function for each pair of data points, in much the same way as the covariance matrix is generated. To produce a scalar update gradient for the hyperparameter, does one just aggregate the gradients of all the derivative covariance matrix values?

EDIT:

After further research, I have found an SGD algorithm for online GP-LVM training on slide 62 found here. So, if we have a latent space vector $\mathbf{x}_{n}$ and the gradient $\frac{\partial L}{\partial \mathbf{x}_{n}}$ is it valid to update all latent space points with this gradient for the current epoch(ignoring the R neighbours, for now)? The gradient would be an $n \times q$ matrix.

Answer 1

Although my answer is late, I thought it may help other people.

Let's first recall the log-likelihood objective function in question (Eq.6, Lawrence2005):

$$L = -\frac{DN}{2}ln 2\pi - \frac{D}{2}ln|K| - \frac{1}{2}tr(K^{-1}YY^{T}),$$

where $$ K = XX^T + \beta^{-1}I$$

The gradient of L w.r.t X is given as follow:

$$\frac{\partial L}{\partial X} = \frac{\partial}{\partial X}(-\frac{DN}{2}ln 2\pi) + \frac{\partial}{\partial X}(- \frac{D}{2}ln|K|) + \frac{\partial}{\partial X}(- \frac{1}{2}tr(K^{-1}YY^{T}))$$

The derivative of the first term $$\frac{\partial}{\partial X}(-\frac{DN}{2}ln 2\pi) = 0$$ as it's a constant.

To obtain the derivative of the second term we use the chain rule, i.e. $$ [f(g(x))]' = f'(g(x))g'(x)$$

So : $$ \frac{\partial}{\partial X}(- \frac{D}{2}ln|K|) = \frac{-D}{2}(\frac{\partial ln|K|}{\partial K})(\frac{\partial K}{\partial X}) = \frac{-D}{2}(\frac{1}{K})(2X) = -DK^{-1}X$$

The same for the last term: $$ \frac{\partial}{\partial X}(- \frac{1}{2}tr(K^{-1}YY^{T})) = \frac{-1}{2}\frac{\partial tr(K^{-1}YY^T)}{\partial K}\frac{\partial K}{\partial X}$$

We have (See MatrixCookbook) $$ \frac{\partial}{\partial Z}tr(AZ^{-1}B) = -(Z^{-1}BAZ^{-1})^T$$

By putting $$A=I, Z=K, B=YY^T$$ we can obtain

$$ \frac{-1}{2}\frac{\partial tr(K^{-1}YY^T)}{\partial K}\frac{\partial K}{\partial X} = \frac{-1}{2}(-1)((K^{-1})^TI^T(YY^T)^T(K^{-1})^T)(2X) = K^{-1}YY^TK^{-1}X$$

We finally obtain $$ \frac{\partial L}{\partial X} = K^{-1}YY^TK^{-1}X - DK^{-1}X $$

For the $\frac{\partial L}{\partial K}$, we need to calculate the derivative of L w.r.t K only, so we just drop the $\frac{\partial K}{\partial X}$ term from the above equation:

$$ \frac{\partial L}{\partial K} = K^{-1}YY^TK^{-1} - DK^{-1} $$

The derivative of L w.r.t $\theta$, i.e. $\frac{\partial L}{\partial \theta}$, is zero simply because $\theta$ is not a variable of L.

Answer 2

I am facing the same today. However I should say the the ultimate gradient is computed by taking the trace, so at the end it is a scalar. Denoting $x_i$ as the i-th component of vector $x$, and L the log-likelihood:

$\frac{\partial L}{\partial x_{i}}= \text{Tr}[\frac{\partial L}{\partial K}\frac{\partial K}{\partial x_{i,j}}]$

So I think we should compute the second term and compute the trace of that matrix product, which can be done by computing the product of the rows of $\frac{\partial L}{\partial K}$ with the columns of $\frac{\partial K}{\partial x_{i,j}}$ and summing up this dots products. I think that those slides do not have the answer we are looking for. Note that this answer answers both of your questions.