0
$\begingroup$

Denote the training data and training target as $X, y$, respectively, now we aim at applying the Gaussian process (GP) for regression. Here, we assume the prior is a GP: $$ p(f(X)) = \mathcal{GP}(m(X), K(X,X)), $$ where $m(\cdot)$ is the mean function and $K(\cdot, \cdot)$ is the covariance matrix with $K$ denoting kernel function. The likelihood is defined as $$ p(y | f, X) = \mathcal{N}(y | f(X), \sigma^2 I). $$ Then, based on the conjugate property of GP prior, the posterior can be derived as $$ p(f(X_{*}) | y, X) = \mathcal{GP}(m_{post}(X_* ), K_{post}(X_*, X_*)), $$ $$ m_{post}(X_*) = m(X_*) + K(X_*, X)(K(X,X)+\sigma^2 I)^{-1}(y - m(X)), $$ $$ K_{post}(X_*, X_*) = K(X_*, K_*) - K(X_*, X)(K(X,X)+\sigma^2 I)^{-1}K(X, X_*). $$ This is the final expression of GP regression (we call it method 1). Alternatively, we can arrive at the same result in another way (we call it method 2), as shown in the following:

We define the joint probability of the function of observed&unobserved data as a GP $$ p(f, f_* | X, X_*) = \mathcal{N}\left( \left[\begin{array}{c} m(X)\\ m(X_*) \end{array}\right], \left[\begin{array}{cc} K(X,X) & K(K, K_* )\\ K(X_*, X) & K(X_*, X_*) \end{array}\right]\right), $$ where $X_*$ denotes the unobserved data. Then the conditional probability of $f_*$ is given by $$ p(f_* | f, X, X_*) = \mathcal{N}( \mathbb{E}[f_* | f, X, X_*], \mathbb{V}[f_* | f, X, X_*]), $$ $$ \mathbb{E}[f_* | f, X, X_*] = m(X_*) + K(X_*, X)(K(X,X)+\sigma^2 I)^{-1}(y - m(X)), $$ $$ \mathbb{V}[f_* | f, X, X_*] = K(X_*, K_*) - K(X_*, X)(K(X,X)+\sigma^2 I)^{-1}K(X, X_*). $$


Obviously, the results of method 1&2 are the same. I can understand method 1 because it is a formal Bayesian perspective. But what is the intuition behind method 2? It seems quite simple than method 1, but it gives the same result. Why does it work? I mean simply given a prior of the joint distribution, and the answer automatedly arises in the posterior. Could anybody give me an intuitive explanation?

PS: I follow the notation of GP in the lecture of Marc Deisenroth. The GP formula in this question is at 1:07:07 of this video, in which you can fully understand the derivation of these two methods.

$\endgroup$
2
  • 1
    $\begingroup$ Something seems off here. You describe method 1 as producing $p(f_* \mid X, y)$ (looks like you forgot to condition on $X_*$ too). This would be the posterior predictive distribution of the latent function values at $X_*$, given the training set $(X,y)$. In contrast, you describe method 2 as producing $p(f_* \mid f, X, X_*)$. This is a different object than in method 1. It's the posterior predictive distribution of the latent function outputs at $X_*$, given observations of $X$ and $f$. But, according to your model, $f$ is latent and only $y$ is observed, so why condition on $f$? $\endgroup$
    – user20160
    Commented Aug 2, 2021 at 11:41
  • $\begingroup$ Finally, the expressions you write in method $2$ depend on $y$ and $\sigma^2$, which doesn't make sense if you're conditioning on $f$ and not $y$. It looks like your original intent for method 2 might've been something along the lines of equations 2.18 and/or 2.21 in GPML? $\endgroup$
    – user20160
    Commented Aug 2, 2021 at 11:41

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.