10
$\begingroup$

I'm trying to wrap my head around this, but when someone says that they used a Gaussian Process, is this not the same as doing linear regression in a feature space defined by the kernel used?

$\endgroup$
2
  • 1
    $\begingroup$ This seems plausible. I am not up on the jargon enough to say for sure, but the answer is probably here: Gaussian Processes for Machine Learning. $\endgroup$
    – GeoMatt22
    Commented Aug 25, 2016 at 2:52
  • $\begingroup$ In particular, see Section 2.1 "Weight Space View" of the GPML textbook by Rasmussen and Williams that @GeoMatt22 linked to. $\endgroup$
    – jII
    Commented Jan 13, 2023 at 2:49

3 Answers 3

5
$\begingroup$

Since an ounce of algebra is equal to a ton of words, let me write some formulas.

Notation

Denote $k( \cdot, \cdot )$ some covariance function, assume we have $m$ observations $(\mathbf x_i, y_i )_{i=1}^m$. Denote

$$ \Sigma = \begin{bmatrix} k( \mathbf x_1 , \mathbf x_1 ) & \dots & k( \mathbf x_1 , \mathbf x_m ) \\ k( \mathbf x_2 , \mathbf x_1 ) & \dots & k( \mathbf x_2, \mathbf x_m ) \\ \vdots & & \vdots \\ k( \mathbf x_m , \mathbf x_1 ) & \dots & k( \mathbf x_m , \mathbf x_m ) \\ \end{bmatrix} \in \mathbb{R}^{m \times m }, \ k(\mathbf x) = \begin{bmatrix} k(\mathbf x, \mathbf x_1 ) \\ \vdots \\ k(\mathbf x, \mathbf x_m ) \end{bmatrix} \in \mathbb{R}^m,\ \mathbf y = \begin{bmatrix} y_1 \\ \vdots \\ y_m \end{bmatrix} \in \mathbb{R}^m $$

and

$$ X = \begin{bmatrix} ------ & \mathbf x_1^t & ------ \\ & \vdots & \\ ------ & \mathbf x_m^t & ------ \end{bmatrix}. $$

Gaussian Process Regression

Gaussian Process Regression (GPR) gives the poserior for $\mathbf x$ as $$ y \sim \mathcal{N} (k^t(\mathbf x) \Sigma^{-1} \mathbf y, k(\mathbf x,\mathbf x) - k^t(\mathbf x) \Sigma^{-1} k(\mathbf x ) ). $$

This arises by assuming $(\mathbf y, y )$ are all jointly Gaussian with zero mean and a covariance structure specified by $k( \cdot, \cdot )$. That's the main idea and the rest is calculations using Schur complements. You would (probably) want to make a prediction based on either the posterior mean or the posterior mode. Luckily, in this case they are the same. You would predict, for a given $\mathbf x$: $$ y^{\star} = k^t(\mathbf x ) \Sigma^{-1} \mathbf y. $$

General Linear Model

A General Linear Model (GLM) arises when you try to find the best linear model to describe observations with a given covariance structure (specified by $\Sigma$). You assume $$ \mathbf y = X \beta + \epsilon, \epsilon \sim \mathcal{N}(0,\Sigma). $$

Then the log-likelihood is

$$ \log p(\mathbf y|X,\beta x) = -\frac{1}{2} (X \beta - \mathbf y )^t\Sigma^{-1} (X\beta - \mathbf y), $$

up to an additive constant. Then the following $\beta^{\star}$ is a Maximum Likelihood Estimator for $\beta$:

$$ \beta^{\star} := \arg \min_{\beta} \| \Sigma^{-1/2} (X\beta - \mathbf y) \|_2^2 \\ = ( (\Sigma^{-1/2} X)^t (\Sigma^{-1/2} X) )^{-1} (\Sigma^{-1/2} X)^t \Sigma^{-1/2}\mathbf y \\ = (X^t \Sigma^{-1} X)^{-1} X^t \Sigma^{-1} \mathbf y. $$

Now, a prediction is made using this linear model as follows:

$$ y^{\star} = \mathbf x^t \beta^{\star} = \mathbf x^t (X^t \Sigma^{-1} X)^{-1} X^t \Sigma^{-1} \mathbf y. $$

Conclusion

The formulas for the posterior mean for GPR and the GLM predictor are clearly different, so this answers your question.

A Few Comments

  1. One key difference is that a GLM does not take into account the covariance between $\mathbf x$ and $\mathbf x_i$, for any $i$. In the GPR model, this information on $\mathbf x$ enters via the vector $k(\mathbf x)$.

  2. Expanding on this point you can think of either one of these models as a weigting scheme used to get from $\mathbf y$ to $y$. In the GLM case, your weights are a linear function of $\mathbf x$ itself. In the GPR case, these weights are a still linear, but now in $k(\mathbf x, \cdot )$! More on this is the book, chapter 2. http://www.gaussianprocess.org/gpml/

  3. The Gaussian Process model is Bayesian. It gives you a posterior distribution (of which you take the mean for prediction). The GLM is frequentist - no posterior distribution, just point estimates (for $\beta^{\star}$ and for $y^{\star}$).

$\endgroup$
1
  • $\begingroup$ Note that OP asks about generalized linear models, which is what GLM most often stands for. GLMs include logistic, probit and linear regression, while general linear regression is just a fancy word for the regular linear regression with vector-valued variables. $\endgroup$ Commented Feb 8, 2023 at 15:34
5
$\begingroup$

No, there are special cases where these two overlap but in general they are different. It is confusing because they are related and can both be used for nonparametric regression. Also the word "kernel" is ambiguous here as there is a distinction between kernel machines and kernel density estimate type nonparametric regression.

An example of overlap is that relevance vector machines (RVMs) which can be seen as type of Bayesian kernelised GLM with sparsity inducing priors, can also be formulated as a Gaussian process. This is described in the Rasmussen & Williams book mentioned in the comment.

Gaussian processes are, strictly speaking, a type of distribution where every finite sample has a joint Gaussian distribution. Nothing says that this distribution needs to be used for regression. Gaussian processes can be used for unsupervised learning, such as Gaussian process latent variable models. Gaussian processes can also be used for optimisation. Kernelised GLMs don't really make sense in either of these contexts.

There are a couple of other differences:

  • GPs require the kernel to be positive semi-definite, kernelised GLMs do not.
  • fitting kernelised GLMs requires parameter estimation, fitting GPs do not.
$\endgroup$
0
1
$\begingroup$

If you are using a Gaussian process for regression and only care about the predictive mean, then it is exactly equivalent to performing kernel ridge regression. I had the same question in mind while reading about these things, therefore decided to compile a short summary of the relationships between Gaussian process and kernel ridge regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.