Sampling from Gaussian Process

Question

I am learning the Gaussian process and feel confused about how three lines were generated in Fig 2.2(A) in the book "Gaussian Process For Machine Learning". As described by the author: "To see this, we can draw samples from the distribution of functions evaluated at any number of points; in detail, we choose a number of input points, $X_∗$, and write out the corresponding covariance matrix using eq. (2.16) elementwise. Then we generate a random Gaussian vector with this covariance matrix".

$$ \operatorname{cov}\big(f(\mathbf{x}_p), f(\mathbf{x}_q) \big) = k(\mathbf{x}_p, \mathbf{x}_q) = \exp\big(-\tfrac{1}{2} |\mathbf{x}_p- \mathbf{x}_q|^2 \big) \tag{2.16} $$

1. What does it mean by drawing samples from the distribution of functions? Does this function refer to $f(x)=W^Tx$ in the linear Bayesian regression, and we are sampling from prior of p(w)? But if using 2.16 and assuming the mean is zero, we don't need this distribution, isn't it?
2. As shown in Fig-2.2(A), there are three samples corresponding to 3 lines, how each line was generated? Are they the "Gaussian vector" described in the text, in other words, at each time, a vector containing all f(x) values on selected x is generated and that is a line, and then generate another vector using the same joint distribution? Thank you

Answer 1

Consider the definition from the book by Rasmussen

Definition 2.1 A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution.

So Gaussian Process is a distribution over functions $f(x_1), f(x_2), \dots, f(x_N)$ such that each of them is considered as a Gaussian random variable, where the relations between them are governed by the mean $m(\mathbf{x})$ and covariance $k(\mathbf{x}, \mathbf{x}')$ functions.

$$ \left[ {\begin{array}{c} f(x_1) \\ f(x_2) \\ \vdots \\ f(x_N) \\ \end{array}} \right] \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma}) = \mathcal{GP}\left(m(\mathbf{x}),\, k(\mathbf{x}, \mathbf{x}')\right) $$

How the plot was generated is described on p. 14 of the book,

The specification of the covariance function implies a distribution over functions. To see this, we can draw samples from the distribution of functions evalu- ated at any number of points; in detail, we choose a number of input points, $X_∗$ and write out the corresponding covariance matrix using eq. (2.16) elementwise. Then we generate a random Gaussian vector with this covariance matrix $$ \mathbf{f}_* \sim \mathcal{N}(\boldsymbol{0}, K(X_*, X_*)) \tag{2.17} $$

So "drawing a sample from Gaussian Process" means sampling from a multivariate normal distribution with mean being the output of the mean function and covariance being the output of the covariance function.

The example above describes how the samples were drawn from the Gaussian Process prior but in the next sections on p. 15-16 the book discusses how to draw the samples from the posterior. The difference is that you simply take the posterior mean and covariance functions there. If you have a non-zero mean function, you just use it in the place of $\boldsymbol{0}$ in 2.17.

The nice smooth lines on the plots are just points from the samples connected with interpolated lines.