I am just trying to gain some intuition behind some of the calculations in Gaussian processes. In chapter 2 of the linked book, equation 2.26 calculates the predictive variance as,
$$ \mathbb{V}[f_*] = k(\mathbf{x}_*, \mathbf{x}_*) - \mathbf{k}_*^\top (\mathbf{K} + \sigma^2\mathbf{I})^{-1}\mathbf{k}_* $$
I think I can interpret the first term $k(\mathbf{x}_*, \mathbf{x}_*)$ intuitively as "what we don't know," as in the maximum variance if we haven't observed any data. If we were using the RBF kernel $\exp{(\frac{|| \mathbf{x} - \mathbf{x}^\prime ||}{2\sigma^2})}$, then $k(\mathbf{x}_*, \mathbf{x}_*)$ would evaluate to 1.
I think I can interpret the second term as "what we have learned from the data," because we know that $\mathbf{k}_*^\top (\mathbf{K} + \sigma^2\mathbf{I})^{-1}\mathbf{k}_*$ is positive due to the PSD kernel and it contains any information which we have learned from the data. Does this sound reasonable so far?
What I am failing to get a good intuition about is the following:
- $\mathbf{k}_*^\top (\mathbf{K} + \sigma^2\mathbf{I})^{-1}\mathbf{k}_*$ is a weighted norm (or Mahalanobis distance) of $\mathbf{k}_*^\top\mathbf{k}_*$, and I fail to grasp why a high value of this means we have low predictive uncertainty.
- $(\mathbf{K} + \sigma^2\mathbf{I})^{-1}\mathbf{k}_*$ is the solution to $(\mathbf{K} + \sigma^2\mathbf{I})\mathbf{z} = \mathbf{k}_*$, so how can I interpret the predictive variance being the dot product $\mathbf{k}_*^\top\mathbf{z}$? What does the dot product with the solution $\mathbf{z}$ represent?
Thanks for any intuition you might have.
