# Derivation of a Simpler Form For Well-explained Points in Newton's Approximation Method For Gaussian Process Binary Classification

I am currently reading Gaussian Process for Machine learning chapter 3.4 on Laplacian Approximation for binary Gaussian Process approximation (classification). I am stuck at section 3.4.1 (page 42) when the author applies Newton's method to approximate the posterior $$p(f|X,y)$$ where $$f$$ is the nuisance gaussian process with covariance matrix $$K$$ and $$y$$ is the class label for design matrix $$X$$. So far, I do understand how Newton's method is applied to iteratively find the maximum of the concave $$\log p(f|X,y)$$:

$$f^{new} = f - (\nabla \nabla \Psi)^{-1}(\nabla \Psi)=(K^{-1}+W)^{-1}(\nabla \log p(y|f) - K^{-1}f)\\ =(K^{-1}+W)^{-1}(Wf+\nabla \log p(y|f))$$ where $$\Psi(f) := \log p(y|f) + \log p(f|X)$$ is the log (unnormalized) posterior and $$W := - \nabla \nabla \log p(y|f).$$

But to gain more intuition about the iterative update, the author considered $$f$$ to contain two subvectors: $$f_1$$ which corresponds points that are NOT well explained, and $$f_2$$ that corresponds to points that are well explained (where $$\partial \log p(y_i | f_i)/\partial f_i$$ and $$W_{ii}$$ are close to zero). The author states in (3.19) that $$$$f_1^{new} = K_{11}(I_{11}+W_{11} K_{11})^{-1}(W_{11} f_1+\nabla \log p(y_1|f_1)),$$$$ $$$$f_2^{new} = K_{21}K_{11}^{-1}f_1^{new}.$$$$

I do understand the first term but do not understand how $$f_2^{new}$$ came about. The author wrote that the second term could be obtained using the partitioned matrix inverse equations. Can someone please help me derive $$f_2^{new}$$ .

After pondering on the problem, I caught my own mistake of treating $$f^{new}_1$$ and $$f^{new}_2$$ independently. I decided to leave my answer here on Cross Validated for anyone who have the same confusion as I did. In this answer I may make a few references to the book GPML, so check it out if anything is unclear.
Firstly, combine $$f^{new}_1$$ and $$f^{new}_2$$ into one vector $$f$$ and rewrite $$(K^{-1} + W)^{-1}$$ as $$K(I+WK)^{-1}$$ (which could be proven equivalent through factorization) in the updating formula. The process onward is rather lengthy so I'll screenshot my calculation and explain it afterward.
In the calculation, $$O$$ represents matrix of all zero. (1) is due to Newton's method updating formula mentioned in the question. (2) is due to the fact that $$W_{ii}$$ and $$\partial \log p(y_i|f_i)/\partial f_i$$ are close to zero for well-explained $$f_i$$s and that $$W$$ is diagonal as mentioned in the GPML book. And (3) is due to the block matrix inversion formula.
We extract the first column of $$f^{new}$$ as $$f^{new}_1$$ and the second as $$f^{new}_2$$. The result of $$f^{new}_2 = K_{21}K_{11}^{-1}f^{new}_1$$ immediately follows as $$K_{11}^{-1}$$ and $$K_{11}$$ cancels out.