2
$\begingroup$

I am attempting to implement the Nonlinear Gaussian Process Latent Variable Model, as per Lawrence 2005 and have the gradient with respect to the kernel as follows(Eq 10 in paper):

$\frac{\partial L}{\partial \mathbf{K}} = K^{-1}YY^{T}K^{-1} -DK^{-1}$

where $K$ is an $n \times n$ covariance matrix(built from an initial latent space representation of $Y$) and $n$ is the number of data points and $Y$ is an $n \times d$ matrix of $d$ dimensional data in the observation space.

I am looking to compute the following gradients:

$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \mathbf{x}_{n}}$

and

$\frac{\partial L}{\partial \mathbf{x}_{n}} = \frac{\partial L}{\partial \mathbf{K}}\frac{\partial \mathbf{K}}{\partial \theta}$

where $\mathbf{x}_{n}$ is the vector of the n'th row of $X$(the initial $n \times q$ latent space - obtained with PCA) and $\theta$ is an arbitrary hyperparameter.

The gradient $\frac{\partial L}{\partial \mathbf{K}}$ is an $n \times n$ matrix, as is the gradient $\frac{\partial \mathbf{K}}{\partial \theta}$.

Two things are not clear at this stage:

1) Taking the gradient w.r.t. a $q$ dimensional feature vector of the covariance matrix yields an $n \times n \times q$ tensor, by evaluating the partial derivative of the covariance function for each pair of points w.r.t the first $q$ dimensional vector argument($a$ for instance). How can one combine this tensor of gradients with $\frac{\partial L}{\partial \mathbf{K}}$ to form a $n \times q$ update matrix for the latent space points?

2) When taking the gradient w.r.t. some arbitrary hyperparameter $\theta$ one obtains an $n \times n$ matrix by evaluating the partial of the covariance function for each pair of data points, in much the same way as the covariance matrix is generated. To produce a scalar update gradient for the hyperparameter, does one just aggregate the gradients of all the derivative covariance matrix values?

EDIT:

After further research, I have found an SGD algorithm for online GP-LVM training on slide 62 found here. So, if we have a latent space vector $\mathbf{x}_{n}$ and the gradient $\frac{\partial L}{\partial \mathbf{x}_{n}}$ is it valid to update all latent space points with this gradient for the current epoch(ignoring the R neighbours, for now)? The gradient would be an $n \times q$ matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.