In sparse approximations of GP for large data set $(X,\mathbf{y})$ with $n$ samples, usually $m$ inducing points are chosen such that the true covariance matrix is approximated by $K_{nn}\to K_{nm}K_{mm}^{-1}K_{mn}$, where $K_{mn}=K_{nm}^T$ is the cross-covariance between inducing point and data points. This results in a lower bound for the marginal likelihood (negative loss function) of (see Titsias 2009) \begin{gather} F_V = \log\mathcal{N}(0,\sigma^2 I_{nn}+K_{nm}K_{mm}^{-1}K_{mn}) - \frac{1}{2\sigma^2} Tr({K_{nn}-K_{nm}K_{mm}^{-1}K_{mn}}) \end{gather} I have already coded this and the code is fast but it sometimes (50% actually!) fails. Let $\Gamma=\sigma^2 I_{nn}+K_{nm}K_{mm}^{-1}K_{mn}$. For evaluating this, we need to calculate (among other things) \begin{equation} \mathbf{y}^T \Gamma^{-1} \mathbf{y} + \log |\Gamma| \end{equation} I used Woodbury equations to convert $\Gamma^{-1} = \sigma^{-2}I_{nn} - \sigma^{-4} K_{nm} \Sigma_{mm}^{-1} K_{mn}$ where $\Sigma_{mm} = K_{mm} + \sigma^{-2} K_{mn}K_{nm}$. But apparently the Cholesky factorization (or inversion) of $\Sigma$ is not well conditioned. Do you know any numerically stable and efficient algorithm to do this (invert $\Gamma$)?

  • $\begingroup$ I have never seen anyone actually inverting $\Gamma$. We want to solve the system. Can you please share a bit of your code? Also given that $\Gamma$ has a ridge parameter in $\sigma^2$ it seems odd that cannot be "somewhat reasonably" conditioned. What are the $\sigma^2$ values when the solution fails? $\endgroup$ – usεr11852 Mar 29 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.