I am currently reading Gaussian Process for Machine learning chapter 3.4 on Laplacian Approximation for binary Gaussian Process approximation (classification). I am stuck at section 3.4.1 (page 42) when the author applies Newton's method to approximate the posterior $p(f|X,y)$ where $f$ is the nuisance gaussian process with covariance matrix $K$ and $y$ is the class label for design matrix $X$. So far, I do understand how Newton's method is applied to iteratively find the maximum of the concave $\log p(f|X,y)$:

$$f^{new} = f - (\nabla \nabla \Psi)^{-1}(\nabla \Psi)=(K^{-1}+W)^{-1}(\nabla \log p(y|f) - K^{-1}f)\\ =(K^{-1}+W)^{-1}(Wf+\nabla \log p(y|f))$$ where $$\Psi(f) := \log p(y|f) + \log p(f|X)$$ is the log (unnormalized) posterior and $$W := - \nabla \nabla \log p(y|f).$$

But to gain more intuition about the iterative update, the author considered $f$ to contain two subvectors: $f_1$ which corresponds points that are NOT well explained, and $f_2$ that corresponds to points that are well explained (where $\partial \log p(y_i | f_i)/\partial f_i$ and $W_{ii}$ are close to zero). The author states in (3.19) that \begin{equation} f_1^{new} = K_{11}(I_{11}+W_{11} K_{11})^{-1}(W_{11} f_1+\nabla \log p(y_1|f_1)), \end{equation} \begin{equation} f_2^{new} = K_{21}K_{11}^{-1}f_1^{new}. \end{equation}

I do understand the first term but do not understand how $f_2^{new}$ came about. The author wrote that the second term could be obtained using the partitioned matrix inverse equations. Can someone please help me derive $f_2^{new}$ .


1 Answer 1


After pondering on the problem, I caught my own mistake of treating $f^{new}_1$ and $f^{new}_2$ independently. I decided to leave my answer here on Cross Validated for anyone who have the same confusion as I did. In this answer I may make a few references to the book GPML, so check it out if anything is unclear.

Firstly, combine $f^{new}_1$ and $f^{new}_2$ into one vector $f$ and rewrite $(K^{-1} + W)^{-1}$ as $K(I+WK)^{-1}$ (which could be proven equivalent through factorization) in the updating formula. The process onward is rather lengthy so I'll screenshot my calculation and explain it afterward. calculation

In the calculation, $O$ represents matrix of all zero. (1) is due to Newton's method updating formula mentioned in the question. (2) is due to the fact that $W_{ii}$ and $\partial \log p(y_i|f_i)/\partial f_i$ are close to zero for well-explained $f_i$s and that $W$ is diagonal as mentioned in the GPML book. And (3) is due to the block matrix inversion formula.

We extract the first column of $f^{new}$ as $f^{new}_1$ and the second as $f^{new}_2$. The result of $f^{new}_2 = K_{21}K_{11}^{-1}f^{new}_1$ immediately follows as $K_{11}^{-1}$ and $K_{11}$ cancels out.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.