I've been reading Gaussian process for machine learning by Rasmussen and Williams and I'm confused by the prior and posterior in the weight space view and function space view.

The prior and posteior are clear in the weight space view as written in Page 9 of the book. $$ p(\mathbf{w}|\mathbf{y}, X) = \frac{p(\mathbf{y}|X,\mathbf{w})p(\mathbf{w})}{p(\mathbf{y}|X)} $$ where $p(\mathbf{w}|\mathbf{y}, X)$ is the posteior on weights, $p(\mathbf{w})$ is the prior, and $p(\mathbf{y}|X)$ is the marginal likelihood (which is used later for model selection).

However, in the function space, the prior and posterior should be defined on a function $f$. Following the formulation above, we have $$ p(f|\mathbf{y}, X) = \frac{p(\mathbf{y}|X,f)p(f)}{p(\mathbf{y}|X)} $$ The problem lies in:

  1. Are $p(f)$ and $p(f|\mathbf{y}, X)$ the correct formulation for the prior and posterior in the function space view?
  2. How to understand the meaning of the distribution of a function like $p(f)$?

1 Answer 1


So if I'm understanding your question correctly, you want to understand how we can translate from the weight space view to the function space view (if possible) and what the differences are in specifying priors/posteriors in those spaces. I think this question is best illustrated with the concrete example of Bayesian linear regression.

Consider the following setup. We have $n$ observations of some data which I summarize into a matrix $\boldsymbol{X} \in \mathbb{R}^{n \times d}$ along with corresponding noisy target values $\boldsymbol{y} \in \mathbb{R}^{n}$. We assume that $\boldsymbol{y}$ is corrupted by some Gaussian additive noise $\boldsymbol{\epsilon} \sim \mathcal{N}(\boldsymbol{0}, \sigma_{\epsilon}^2\boldsymbol{1}_{n \times n})$ and we consider the following model: $${y_i} = f(\boldsymbol{x}_i) + \epsilon_i $$

  1. $\textit{Weight Space View: }$ In the weight space view, we assume that $f$ is a parametrized function, e.g $f(\boldsymbol{x}) = f_{\boldsymbol{w}}(\boldsymbol{x})$ for some parameters $\boldsymbol{w}$. In this view, we place the prior on the parameters of the model, for sake of simplicity we assume the prior: $$p(\boldsymbol{w}) = \mathcal{N}(\boldsymbol{0}, \boldsymbol{\Sigma}_0)$$ Due to the additive noise, the likelihood is also Gaussian: $$p(\boldsymbol{y}|\boldsymbol{w}, \boldsymbol{X}) = \mathcal{N}(\boldsymbol{X}\boldsymbol{w}, \sigma_{\epsilon}^2\boldsymbol{1}_{n \times n})$$ I won't go into the details of deriving the posterior as this is anyways not what you asked but as shown in the book we obtain $$p(\boldsymbol{w}|\boldsymbol{X}, \boldsymbol{y}) = \mathcal{N}(\boldsymbol{\mu}_{\text{post}}, \boldsymbol{\Sigma}_{\text{post}})$$ with the following posterior statistics: $$\boldsymbol{\mu}_{\text{post}} = \frac{1}{\sigma^2}\boldsymbol{A}_{\boldsymbol{w}}^{-1}\boldsymbol{X}\boldsymbol{y} \hspace{3mm} \text{ and } \hspace{3mm} \boldsymbol{\Sigma}_{\text{post}} = \boldsymbol{A}_{\boldsymbol{w}}^{-1}$$ where we define $\boldsymbol{A}_{\boldsymbol{w}} = \frac{1}{\sigma^2_{\epsilon}}\boldsymbol{X}\boldsymbol{X}^{T} + \boldsymbol{\Sigma}_{\boldsymbol{w}}^{-1}$

  2. $\textit{Function Space View: }$ Let's now try to translate the above example into the function space view. I hope this will help your intuition! As you remarked correctly, now we need to specify a prior in function space, so directly on $f$ without resorting to any sort of parametrization $\boldsymbol{w}$. This means that for every $\boldsymbol{x} \in \mathbb{R}^{d}$ we need to characterize the distribution of $f(\boldsymbol{x})$ as well as the covariance structure $K(\boldsymbol{x}, \boldsymbol{x}')=\text{cov}\left(f(\boldsymbol{x}), f(\boldsymbol{x}')\right)$ for any other $\boldsymbol{x}' \in \mathbb{R}^{d}$. This is where the Gaussian process enters. A Gaussian process (very intuitively stated) is an infinite-dimensional analog to the multivariate Gaussian distribution we used before and hence allows you to specify the distribution of $f$ at every point $\boldsymbol{x}$, something you can't do with an ordinary finite Gaussian distribution! There are some mathematical subtleties however that I am conveniently ignoring here as they do not add to the story. This infinite Gaussian distribution is characterized completely by the following property: $$\forall \boldsymbol{x}_1, \dots,\boldsymbol{x}_m \in \mathbb{R}^{d}: \left(f(\boldsymbol{x}_1), \dots f(\boldsymbol{x}_m)\right) \sim \mathcal{N}(\boldsymbol{\mu}, K(\boldsymbol{X}, \boldsymbol{X}))$$ where $K(\boldsymbol{X}, \boldsymbol{X})_{ij} = K(\boldsymbol{x}_i, \boldsymbol{x}_j) = \text{cov}\left(f(\boldsymbol{x}_i), f(\boldsymbol{x}_j)\right)$. A Gaussian process is hence defined via its "marginals" on finite subsets of $\mathbb{R}^{d}$ and determined through the specification of the two functions $\mu(\boldsymbol{x})=\mathbb{E}[f(\boldsymbol{x})]$ and $K(\boldsymbol{x}, \boldsymbol{x}') = \text{cov}\left(f(\boldsymbol{x}), f(\boldsymbol{x}')\right)$. These two functions are our prior "design-choices", much like we can specify a prior mean $\boldsymbol{\mu}_0$ and prior covariance $\boldsymbol{\Sigma}_0$ in the weight space view. We usually denote this as $\mathcal{GP}(\mu, K)$. Let's now try to phrase our linear regression problem as a Gaussian process! Observe that for a fixed $\boldsymbol{x}$, $$f_\boldsymbol{w}(\boldsymbol{x})=\boldsymbol{w}^{T}\boldsymbol{x} = \sum_{i=1}^{d}w_i x_i$$ is distributed as a Gaussian by the definition of joint Gaussianity of $\boldsymbol{w}$. Moreover we find that $$\mu_0(\boldsymbol{x}) = \mathbb{E}[f_\boldsymbol{w}(\boldsymbol{x})] = {0}$$ as well as the covariance structure $$K_0(\boldsymbol{x},\boldsymbol{x}') = \mathbb{E}[f_\boldsymbol{w}(\boldsymbol{x})f_\boldsymbol{w}(\boldsymbol{x}')] = \boldsymbol{x}^{T}\boldsymbol{\Sigma}_0\boldsymbol{x}'$$ We can hence see that imposing a prior on the parameters $\boldsymbol{w}$ can also be translated into a prior directly on the function $f_{\boldsymbol{w}}$ as $$f_{\boldsymbol{w}} \sim \mathcal{GP}(0, K_0)$$ Notice however that we of course lost the degree of freedom to specify whatever $\mu$ and $K$ we wanted!

I hope this illustrates the duality between Bayesian regression and Gaussian process regression.

To answer the other questions, usually we want to avoid to talk about the "entire" distribution $p(f)$ but rather work with the marginals $p(f(\boldsymbol{x}_1), \dots, f(\boldsymbol{x}_n))$. Moreover, we usually directly move to the predictive distribution $$p(f^{*}|\boldsymbol{y},\boldsymbol{X})$$ where $f^{*}$ corresponds to a new test point $\boldsymbol{x}$ by using nice properties of the Gaussian distribution which allow to easily derive $p(f^{*}|\boldsymbol{y},\boldsymbol{X})$ from the joint distribution $p(f^{*}, \boldsymbol{y}|\boldsymbol{X})$. This is slightly confusing and in some sense hides the entire Bayesian mechanism for Gaussian process regression. In those derivations we are not using Bayes rule at all (explicitly)! This only becomes obvious if you view the problem in weight space.

As a final remark, notice that reformulating Gaussian process regression to Bayesian regression almost always works, revealing their Bayesian nature more explicitly. This is due to the fact that we can always write the given kernel $K$ as $$K(\boldsymbol{x}, \boldsymbol{x}') = \Phi(\boldsymbol{x})^{T} \Phi(\boldsymbol{x}')$$ where $\Phi: \mathbb{R}^{d} \xrightarrow[]{}\mathbb{R}^{M}$ is the corresponding feature map. Then we can parametrize the corresponding Gaussian process function $f$ equivalently as $$f_{\boldsymbol{w}}(\boldsymbol{x}) =\boldsymbol{w}^{T}\Phi(\boldsymbol{x}) $$ Notice the change of dimensionality: $\boldsymbol{w} \in \mathbb{R}^{M}$. If $K$ induces a infinite-dimensional feature representation (such as the RBF kernel) one has to be a bit careful but one can essentially work with an infinite sum $$f_{\boldsymbol{w}}(\boldsymbol{x})=\sum_{i=1}^{\infty}w_i \Phi_i(\boldsymbol{x})$$ turning $\boldsymbol{w}$ also into a Gaussian process!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.