Laplace Approximation of Gaussian Classifier step clarification I'm reading Gaussian Processes for Machine Learning (Rasmussen and Williams) and trying to understand an equation. In chapter 3 section 4 they're going over the derivation of the Laplace Approximation for a binary Gaussian Process classifier.
$\mathbf{f}^{\text{new}} = \mathbf{f} - (\nabla \nabla \Psi)^{-1}\nabla \Psi = \mathbf{f} + (K^{-1} + W)^{-1}(\nabla \log p(\mathbf{y} \mid \mathbf{f} ) - K^{-1}\mathbf{f}) $
$= (K^{-1} + W)^{-1}(W\mathbf{f} + \nabla \log p(\mathbf{y} \mid \mathbf{f} ) ) \tag{3.18}$
Now, I see where they went from step 1 to step 2 using equations $\text{3.13}$ and $\text{3.14}$ since we have
$ \nabla \Psi(f) =  \nabla \log p(\mathbf{y} \mid \mathbf{f} ) - K^{-1}\mathbf{f}  \tag{3.13}$
and
$ \nabla \nabla \Psi(f) =  \nabla \nabla \log p(\mathbf{y} \mid \mathbf{f} ) - K^{-1}  = -W - K^{-1}\tag{3.14}$
So step 2 replaces $\nabla \Psi(f)$ and $\nabla \nabla \Psi(f) $ and pulls out the negative it looks like but I don't get where $\mathbf{f}$ disappeared to in the next step.
Any help would be great, thanks.
 A: This is definitely a bit opaque! I think I worked out how to derive that step, but it may not be the most direct way. Actually, I have two ways of looking at it.
The first is just a check. Both expressions share the same $\nabla \log p(\mathbf{y}|\mathbf{f})$ term, so we are left with having to show that
$$
(K^{-1} + W)^{-1} W \mathbf{f} = \mathbf{f} - (K^{-1} + W)^{-1} K^{-1} \mathbf{f}
$$
To check that this is correct, multiply both sides by $(K^{-1} + W)$, and you'll see that we get $W\mathbf{f} = W\mathbf{f}$, so this is indeed correct. However, this doesn't tell you how to get from one to the other!
The way I found to do that is to use the Woodbury matrix identity: https://en.wikipedia.org/wiki/Woodbury_matrix_identity . The identity is:
$$
(A + UCV)^{-1} = A^{-1} - A^{-1} U (C^{-1} + V A^{-1} U)^{-1} V A^{-1}.
$$
Now, setting $A = W$, $U = I$, $C = K^{-1}$, $V = I$, we get that
$$
(W + K^{-1}) = W^{-1} - W^{-1}(K + W^{-1})^{-1} W^{-1} \\
= W^{-1} - (WK + I)^{-1} W^{-1} \\
= W^{-1} - (W + K^{-1})^{-1}K^{-1} W^{-1}
$$
and so you can see that multiplying by $W\mathbf{f}$ will result in the right hand side of the first equation, as required.
As I mentioned, I don't think this is particularly neat but I hope it shows you one way to derive this. But perhaps someone knows of a more direct way!
A: When you want to transform multiple additive terms into multiple multiplicative terms, you factor. We see $(K^{-1}+W)^{-1}$ on both sides of
$$
 \mathbf{f} - (K^{-1} + W)^{-1} K^{-1} \mathbf{f} = (K^{-1} + W)^{-1} W \mathbf{f} 
,$$
so factoring that out might be a good idea. But looking at the additive term, we don't see $(K^{-1}+W)^{-1}$ on the left side, so we first multiply $(K^{-1}+W)$ with $\mathbf{f}$ before factoring so that when we multiply the factorization out, $(K^{-1}+W)$ and $(K^{-1}+W)^{-1}$ cancels out to give us $\mathbf{f}$. Doing so, we yield:
$$
\mathbf{f} - (K^{-1} + W)^{-1} K^{-1} \mathbf{f} 
= (K^{-1} + W)^{-1} ((K^{-1} + W)\mathbf{f} - K^{-1} \mathbf{f})\\
= (K^{-1} + W)^{-1} (K^{-1}\mathbf{f} + W\mathbf{f} - K^{-1} \mathbf{f})\\
= (K^{-1} + W)^{-1} W \mathbf{f} 
$$
