I have this confusion related to the benefits of Gaussian processes. I mean comparing it to simple linear regression, where we have defined that the linear function models the data.

However, in Gaussian processes we define the distribution of the functions means we don't specifically define that the function should be linear. We can define a prior over the function which is the Gaussian prior which defines features like how much smooth the function should be and all.

So we don't have to explicitly define what the model should be. However, I have questions. We do have marginal likelihood and using it we can tune the covariance function parameters of the gaussian prior. So this is similar to defining the type of function that it should be isn't it.

It boils down to the same thing defining the parameters even though in GP they are hyperparameters. For eg in this paper. They have defined that the mean function of the GP is something like

$$m(x) = ax ^2 + bx + c \quad \text{i.e. a second order polynomial.}$$

So definitely the model/function is defined isn't it. So what's the difference in defining the function to be linear like in the LR.

I just didn't get what the benefit is of using GP


2 Answers 2


Let's recall some formulas about the Gaussian process regression. Suppose that we have a sample $D = (X,\mathbf{y}) = \{(\mathbf{x}_i, y_i)\}_{i = 1}^N$. For this sample loglikelihood has the form: $$ L = -\frac12 \left( \log |K| + \mathbf{y}^T K^{-1} \mathbf{y}\right), $$ where $K = \{k(\mathbf{x}_i, \mathbf{x}_j)\}_{i, j = 1}^N$ is the sample covariance matrix. There $k(\mathbf{x}_i, \mathbf{x}_j)$ is a covariance function with parameters we tune using the loglikelihood maximization. The prediction (posterior mean) for a new point $\mathbf{x}$ has the form: $$ \hat{y}(\mathbf{x}) = \mathbf{k} K^{-1} \mathbf{y}, $$ there $\mathbf{k} = \{k(\mathbf{x}, \mathbf{x}_i)\}_{i = 1}^N$ is a vector of covariances between new point and sample points.

Now note that Gaussian processes regression can model exact linear models. Suppose that covariance function has the form $k(\mathbf{x}_i, \mathbf{x}_j) = \mathbf{x}_i^T \mathbf{x}_j$. In this case prediction has the form: $$ \hat{y}(\mathbf{x}) = \mathbf{x}^T X^T (X X^T)^{-1} \mathbf{y} = \mathbf{x}^T (X^T X)^{-1} X^T \mathbf{y}. $$ The identity is true in case $(X X^T)^{-1}$ is nonsingular which is not the case, but this is not a problem in case we use covariance matrix regularization. So, the rightest hand side is the exact formula for linear regression, and we can do linear regression with Gaussian processes using proper covariance function.

Now let's consider a Gaussian processes regression with another covariance function (for example, squared exponential covariance function of the form $\exp \left( -(\mathbf{x}_i - \mathbf{x}_j)^T A^{-1} (\mathbf{x}_i - \mathbf{x}_j) \right)$, there $A$ is a matrix of hyperparameters we tune). Obviously, in this case posterior mean is not a linear function (see image).

enter image description here.

So, the benefit is that we can model nonlinear functions using a proper covariance function (we can select a state-of-the-art one, in most cases squared exponential covariance function is a rather good choice). The source of nonlinearity is not the trend component you mentioned, but the covariance function.

  • 3
    $\begingroup$ I would say this is only one benefit of GP with is also shared with other kernel methods. Being probabilistic and coming from the Bayesian frameworkis another advantage of GP. $\endgroup$
    – Leila
    Jan 12, 2017 at 2:13

For me the biggest advantage of Gaussian Processes is the inherent ability to model the uncertainty of the model. This is incredibly useful because, given the expected value of a function and the corresponding variance I can define a metric (i.e. an Aquisition function) that can tell me e.g. what's the point $x$ that, I should evaluate my underlying function $f$ at, that will result in the highest (in expectation) value of $f(x)$. This forms the basis of Bayesian Optimization.

You probably know the exploration vs. exploitation trade-off. We want to find a $max$ of some function $f$ (which is often expensive to evaluate) and so we need to be frugal about which $x$ we select to evaluate $f$. We will probably want to look at places near the points where we know that the function has a high value (exploitation) or at the points where we have no idea about the function's value (exploration). Gaussian Processes give us the necessary information to make a decision concerning the next evaluation: mean value $\mu$ and covariance matrix $\Sigma$ (uncertainty), allowing for e.g. optimizing expensive black-box functions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.