Difference between Gaussian process regression and other regression techniques (say linear regression)

Question

I am confused about the differences in the regression techniques available.

Take for example, linear regression. In this case, we construct a model $y = \beta^Tx + \epsilon$ where $\epsilon \sim N(0,\sigma^2)$. In a sense, $y$ then becomes a "Gaussian process" whose mean is $\beta^Tx$ while its covariance function is $k(x,x')=\sigma^2 \mathbb{1}_{x = x'}$.

On the other hand, Gaussian process regression (as in the GP for ML book) is modeled as $y \sim N(m(x),k(x,x'))$ for some kernel/covariance function $k(x,x')$. This type of model is then used to interpolate a given set of data using basis functions which result from the covariance function.

The main difference I see is that the linear regression (or really, generalized regression of this form), creates a model that does not pass through the data points but rather finds the model which has the "best fit". Of course, the predictor need not be linear. On the other hand, Gaussian process regression uses conditioning on Gaussian vectors to find a model that actually passes through the data points.

With this in mind:

• What really is Gaussian process regression? Can the linear regression with normally distributed $\epsilon$ still be considered Gaussian process regression, as opposed to the Gaussian process regression which interpolates the data (i.e. kriging)? I am confused because Wikipedia shows that Gaussian process regression need not interpolate the data points as shown in the figure here: link.

Can someone help me clarify this confusion?

Answer 1

A Gaussian Process doesn't have to perfectly interpolate between points, as that Wikipedia link shows; it all depends on the covariance function that you use.

For example, consider the GP of the form $X \sim \mathcal N(0, \Sigma_{k_t})$, where $X$ is a vector of a "dependant variables", and $\Sigma_{k_t}$ is a covariance matrix, where every element $\Sigma_{ij} = k(t_i, t_j)$ for some kernel function $k$, and a set of points of the "independent variable" $t$.

If you specify a kernel with the following property: $Cor(x_i, x_j) \to 1$ as $||t_i - t_j|| \to 0$, notice that you are enforcing continuity. Hence, if you simply use such a kernel, for example, the RBF, it must pass through all the points as there's no "noise" here at all.

Instead, if you decide to specify a kernel that does account for noise, for example: $k(t_i, t_j) = RBF(t_i, t_j) + \sigma^2 \mathcal I(t_i =t_j)$ (the WhiteKernel in scikit-learn, also known as the White Noise kernel), then notice that, even if the two $t$s are close, their correlation isn't 1, i.e. there's some noise here. So the function is not expected to be continuous.

In fact, you can interpret using such a kernel as the traditional smooth RBF GP but with a noise term added on top:

$$X \sim \mathcal N(0, \Sigma_{RBF} + \sigma^2 \mathcal I) $$ $$\stackrel d= \mathcal N(0, \Sigma_{RBF}) + \mathcal N(0, \sigma^2 \mathcal I) $$ $$\Rightarrow X = \bar X +\epsilon$$

... where $\bar X$ is now a continuous GP. Notice how similar this is to the linear regression equation - the only difference really is that you're replacing the mean of the linear regression (which is a parametric line) to a non-parametric GP.

Answer 2

In the process part of the gaussian process regression name, there is a notion of continuity which is constrained by the use of a covariance kernel. The underlying assumption is that the variable is spatially auto-correlated, which means that the knowledge of the outcome at some point will give you information at the nearest locations. This is the main difference between a gaussian process and a simple gaussian variable.

Regarding regression, the main obvious difference between gaussian process regression and "classic" regression techniques, is that you do not force an analytical formula for the predictor, but a covariance structure for the outcomes.

Gaussian process regression is very flexible with respect to interpolation. You can make it an exact interpolator, as long as you don't have two different outcomes for the same input. You can relieve it from the exact interpolation constraint either by bounding the covariance kernel hyperparameters values, or, more usually, by adding some white noise to the covariance kernel.

In your example of linear regression, you have $y = \beta^Tx + \epsilon$ with $\epsilon \sim \mathcal{N}(0,\sigma^2)$. I think you can call it a kind of gaussian process regression as soon as you assume that $\epsilon(x)$ is a gaussian process itself, and not just some random gaussian variable (i.e. you assume that $\epsilon$ is spatially auto-correlated). In the general case of linear regression, the $\epsilon$ term is just assumed to be a white noise, and therefore you cannot call it gaussian process regression.

Using both classical regression and gaussian process regression is a quite usual meta-modelling technique in the industry. The basic idea is to first make a fit by a classical technique (any analytical regression, or a deterministic computer code) which is determined by the prior knowledge of the actual process. Then, with experimental data, you can fit the discrepancy between the model and reality, assuming it follows a gaussian process. This will account for the fact that your analytical model is a simplified version of the true process, along with the measurement uncertainties. This is often referred to as bayesian calibration by gaussian processes.