In the following, scalars are denoted with italic lowercases (e.g., $k,\, b_f$), vectors with bold lowercases (e.g., $\mathbf{s},\, \mathbf{x}_i$), and matrices with italic uppercases (e.g., $W_f$).


A Gaussian process (GP) is defined as a collection of random variables, any finite number of which have a joint Gaussian distribution. A GP $f(\mathbf{x})$ is completely specified by its mean function $m(\mathbf{x})$ and covariance function $k(\mathbf{x}, \mathbf{x}'),$ also called kernel, defined as: \begin{align*} m(\mathbf{x}) &= \mathbb{E}[f(\mathbf{x})], \\ k(\mathbf{x}, \mathbf{x}') &= \mathbb{E}[(f(\mathbf{x})-m(\mathbf{x}))(f(\mathbf{x}')-m(\mathbf{x}'))]. \end{align*}


  • ${X = (\mathbf{x}_1,\dotsc,\mathbf{x}_q)}$ be the training inputs
  • $ {\mathbf{f} = (f(\mathbf{x}_1)\dotsc,f(\mathbf{x}_q))}$ be the training outputs
  • $X^* = (\mathbf{x}_{q+1},\dotsc,\mathbf{x}_{s})$ be the test inputs
  • $\mathbf{f}^* = (f(\mathbf{x}_{q+1})\dotsc,f(\mathbf{x}_{s}))$ be the test outputs.

Assume the training set isn't contaminated with samples from the test set, i.e.: $X \cup X^* = \mathcal{X},$ and $X \cap X^* = \emptyset$.

Note that $\mathbf{f}$ is known, and $\mathbf{f}^*$ is unknown. The goal is to find the distribution of $\mathbf{f}^*$ given $X^*, X$ and $\mathbf{f}$.


The joint distribution of $\mathbf{f}$ and $\mathbf{f}^*$ according to the prior is

\begin{equation*} \begin{bmatrix} \mathbf{f} \\ \mathbf{f}^* \end{bmatrix} \sim \mathcal{N} \left( \begin{bmatrix} \mathbf{m} \\ \mathbf{m}^* \end{bmatrix}, ~ \begin{bmatrix} K(X,X) & K(X,X^*) \\ K(X^*,X) & K(X^*,X^*) \\ \end{bmatrix} \right) \end{equation*} where $\mathbf{m}\,,\,\mathbf{m}^*$ is a vector of the means evaluated at all training and test points respectively, and $K(X,X^*)$ denotes the $q \times q^*$ matrix of the covariances evaluated at all pairs of training and test points, and similarly for $K(X,X),\, K(X^*,X)$ and $K(X^*,X^*)$.

Each element of the matrix $K(X,X^*)$ are denoted by $k(\mathbf{x}, \mathbf{x}')$. Ideally one should have $k(\mathbf{x}, \mathbf{x}') = \mathbb{E}[(f(\mathbf{x})-m(\mathbf{x}))(f(\mathbf{x}^*)-m(\mathbf{x}^*))]$ but this isn't feasible since one does not have access to $f(\mathbf{x}^*)$ and $m(\mathbf{x}^*)$. As a result, one resorts to some approximation of $k$ with some kernels taking only $\mathbf{x}$ and $\mathbf{x}^*$ as input e.g.:

  • Linear: $k(\mathbf{x}, \mathbf{x}') = \mathbf{x}^T \mathbf{x}'$
  • Cubic: $k(\mathbf{x}, \mathbf{x}') = 3 \left (\left (\mathbf{x}^T \mathbf{x}' \right )^2 + 2\left ( \mathbf{x}^T \mathbf{x}' \right )^3 \right )$
  • Absolute exponential: $k(\mathbf{x}, \mathbf{x}') = e^{|\mathbf{x}-\mathbf{x}'|} $
  • Squared exponential: $k(\mathbf{x}, \mathbf{x}') = e^{-0.5|\mathbf{x}-\mathbf{x}'|^2} $
  • etc

Conditioning the joint Gaussian prior on the observations yields $\mathbf{f}^* | X^*, X, \mathbf{f} \, \sim \, \mathcal{N} (\mathbf{\mu}, \mathbf{\Sigma})$ where

\begin{align} \mathbf{\mu} &= \mathbf{m}^* - K(X^*,X) K(X,X)^{-1}(\mathbf{f} - \mathbf{m}), \\ \mathbf{\Sigma} &= K(X^*,X^*) - K(X^*,X) K(X,X)^{-1} K(X,X^*). \notag \end{align}


How am I supposed to compute the vector of the means $\mathbf{m}^*$?


1 Answer 1


Take a look at pages 16-17 of the Rasmussen and Williams GPML book: http://www.gaussianprocess.org/gpml/chapters/RW.pdf

We can assume without loss of generality that the mean functions $\mathbf{m}$ and $\mathbf{m^*}$ are 0. I also think that you have an extra negative sign in your posterior, so then the joint distribution is:

\begin{equation*} \begin{bmatrix} \mathbf{f} \\ \mathbf{f}^* \end{bmatrix} \sim \mathcal{N} \left( \begin{bmatrix} \mathbf{0} \\ \mathbf{0} \end{bmatrix}, ~ \begin{bmatrix} K(X,X) & K(X,X^*) \\ K(X^*,X) & K(X^*,X^*) \\ \end{bmatrix} \right) \end{equation*}

and the posterior is

\begin{equation*}\mathbf{\mu} = K(X^*,X) K(X,X)^{-1}\mathbf{f}\end{equation*} \begin{equation*}\mathbf{\Sigma} = K(X^*,X^*) - K(X^*,X) K(X,X)^{-1} K(X,X^*)\end{equation*}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.