I'm trying to wrap my head around this, but when someone says that they used a Gaussian Process, is this not the same as doing linear regression in a feature space defined by the kernel used?


Since an ounce of algebra is equal to a ton of words, let me write some formulas.


Denote $k( \cdot, \cdot )$ some covariance function, assume we have $m$ observations $(\mathbf x_i, y_i )_{i=1}^m$. Denote

$$ \Sigma = \begin{bmatrix} k( \mathbf x_1 , \mathbf x_1 ) & \dots & k( \mathbf x_1 , \mathbf x_m ) \\ k( \mathbf x_2 , \mathbf x_1 ) & \dots & k( \mathbf x_2, \mathbf x_m ) \\ \vdots & & \vdots \\ k( \mathbf x_m , \mathbf x_1 ) & \dots & k( \mathbf x_m , \mathbf x_m ) \\ \end{bmatrix} \in \mathbb{R}^{m \times m }, \ k(\mathbf x) = \begin{bmatrix} k(\mathbf x, \mathbf x_1 ) \\ \vdots \\ k(\mathbf x, \mathbf x_m ) \end{bmatrix} \in \mathbb{R}^m,\ \mathbf y = \begin{bmatrix} y_1 \\ \vdots \\ y_m \end{bmatrix} \in \mathbb{R}^m $$


$$ X = \begin{bmatrix} ------ & \mathbf x_1^t & ------ \\ & \vdots & \\ ------ & \mathbf x_m^t & ------ \end{bmatrix}. $$

Gaussian Process Regression

Gaussian Process Regression (GPR) gives the poserior for $\mathbf x$ as $$ y \sim \mathcal{N} (k^t(\mathbf x) \Sigma^{-1} \mathbf y, k(\mathbf x,\mathbf x) - k^t(\mathbf x) \Sigma^{-1} k(\mathbf x ) ). $$

This arises by assuming $(\mathbf y, y )$ are all jointly Gaussian with zero mean and a covariance structure specified by $k( \cdot, \cdot )$. That's the main idea and the rest is calculations using Schur complements. You would (probably) want to make a prediction based on either the posterior mean or the posterior mode. Luckily, in this case they are the same. You would predict, for a given $\mathbf x$: $$ y^{\star} = k^t(\mathbf x ) \Sigma^{-1} \mathbf y. $$

General Linear Model

A General Linear Model (GLM) arises when you try to find the best linear model to describe observations with a given covariance structure (specified by $\Sigma$). You assume $$ \mathbf y = X \beta + \epsilon, \epsilon \sim \mathcal{N}(0,\Sigma). $$

Then the log-likelihood is

$$ \log p(\mathbf y|X,\beta x) = -\frac{1}{2} (X \beta - \mathbf y )^t\Sigma^{-1} (X\beta - \mathbf y), $$

up to an additive constant. Then the following $\beta^{\star}$ is a Maximum Likelihood Estimator for $\beta$:

$$ \beta^{\star} := \arg \min_{\beta} \| \Sigma^{-1/2} (X\beta - \mathbf y) \|_2^2 \\ = ( (\Sigma^{-1/2} X)^t (\Sigma^{-1/2} X) )^{-1} (\Sigma^{-1/2} X)^t \Sigma^{-1/2}\mathbf y \\ = (X^t \Sigma^{-1} X)^{-1} X^t \Sigma^{-1} \mathbf y. $$

Now, a prediction is made using this linear model as follows:

$$ y^{\star} = \mathbf x^t \beta^{\star} = \mathbf x^t (X^t \Sigma^{-1} X)^{-1} X^t \Sigma^{-1} \mathbf y. $$


The formulas for the posterior mean for GPR and the GLM predictor are clearly different, so this answers your question.

A Few Comments

  1. One key difference is that a GLM does not take into account the covariance between $\mathbf x$ and $\mathbf x_i$, for any $i$. In the GPR model, this information on $\mathbf x$ enters via the vector $k(\mathbf x)$.

  2. Expanding on this point you can think of either one of these models as a weigting scheme used to get from $\mathbf y$ to $y$. In the GLM case, your weights are a linear function of $\mathbf x$ itself. In the GPR case, these weights are a still linear, but now in $k(\mathbf x, \cdot )$! More on this is the book, chapter 2. http://www.gaussianprocess.org/gpml/

  3. The Gaussian Process model is Bayesian. It gives you a posterior distribution (of which you take the mean for prediction). The GLM is frequentist - no posterior distribution, just point estimates (for $\beta^{\star}$ and for $y^{\star}$).


No, there are special cases where these two overlap but in general they are different. It is confusing because they are related and can both be used for nonparametric regression. Also the word "kernel" is ambiguous here as there is a distinction between kernel machines and kernel density estimate type nonparametric regression.

An example of overlap is that relevance vector machines (RVMs) which can be seen as type of Bayesian kernelised GLM with sparsity inducing priors, can also be formulated as a Gaussian process. This is described in the Rasmussen & Williams book mentioned in the comment.

Gaussian processes are, strictly speaking, a type of distribution where every finite sample has a joint Gaussian distribution. Nothing says that this distribution needs to be used for regression. Gaussian processes can be used for unsupervised learning, such as Gaussian process latent variable models. Gaussian processes can also be used for optimisation. Kernelised GLMs don't really make sense in either of these contexts.

There are a couple of other differences:

  • GPs require the kernel to be positive semi-definite, kernelised GLMs do not.
  • fitting kernelised GLMs requires parameter estimation, fitting GPs do not.

If you are using a Gaussian process for regression and only care about the predictive mean, then it is exactly equivalent to performing kernel ridge regression. I had the same question in mind while reading about these things, therefore decided to compile a short summary of the relationships between Gaussian process and kernel ridge regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.