5
$\begingroup$

I had a discussion today with someone saying that Gaussian Processes are linear models. I don't see in which sense this may be correct. To be clear, here the definition of a linear model is the usual one, i.e., a model which is linear in the parameters. Thus,

$y=\beta_0+\beta_1x+\beta_2\sin(x)+\epsilon$

and

$y=\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x}+\epsilon$

are linear, and

$y=\beta_0+\beta_1x+\beta_2\exp({\beta_3x})+\epsilon$

is not.

For simplicity, let's consider a Squared Exponential covariance function, and assume that the correlation length, the signal variance and the noise variance are known. Given a design matrix $X$ and corresponding response vector $\mathbf{y}$, the GP prediction at a new prediction point $\mathbf{x}^*$ is

$$\hat{y}(\mathbf{x}^*)=\mathbf{k}_*^T(K+\sigma I)^{-1}\mathbf{y}$$

Now, this estimator is clearly a nonlinear function of $X$ and a linear function of $\mathbf{y}$. The other person insisted that $\mathbf{y}$ is the parameter vector of this model, and thus the model is linear. I don't think this makes any sense: it would mean that the number of parameters of the model depends on the sample size. I think we can at most say that the estimator is a linear function of $\mathbf{y}$, but surely not the statistical model underlying Gaussian Process Regression. Do you agree?

$\endgroup$
  • 2
    $\begingroup$ Have you noticed that the OLS estimator $(X^\prime X)^{-}X^\prime y$ is highly nonlinear in $X$, too? $\endgroup$ – whuber Oct 31 '17 at 19:43
  • 1
    $\begingroup$ @whuber sure, and it's also linear in $\mathbf{y}$, like the GPR estimator. However, I was talking about linearity of the model, not the estimator. I guess the other person confounded the two things. $\endgroup$ – DeltaIV Oct 31 '17 at 21:58
  • 1
    $\begingroup$ Interestingly, a GP regression model is equivalent to (linear) ridge regression if you use a linear kernel. $\endgroup$ – Kevin Yang Nov 1 '17 at 1:13
  • $\begingroup$ @KevinYang true, but a linear kernel is a degenerate kernel, in that its eigenexpansion is finite (and thus the covariance function is only semidefinite positive, instead than definite positive). Let's consider just strictly positive definite covariance functions: do you think GP regression is a linear model or not? If it is, and you can show me which are the model parameters I would accept it as an answer. However, if the parameters depend on the data set, I don't think we can talk of a linear model...ideally the model is defined independently of the data set. $\endgroup$ – DeltaIV Nov 1 '17 at 11:25
3
$\begingroup$

I think the technically correct term to use here is that GP regression is a linear smoother, i.e. its predictions are a linearly weighted combination of past observed outputs. This does not make the model as such linear. For that to be true, the predictions must be a linear function of the inputs. This is only the case with GPs if you use a linear covariance function.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

The statistical definition of a model being linear is that the model must be linear in its parameters. Gaussian Process Regression can be defined by using either the function-space view or the weight-space view to reach the formula for the posterior mean and posterior variance.

If we see the weight-space view, we can clearly see that Gaussian Process Regression is indeed a linear model with non linear functions of the inputs. The model is defined as a bayesian linear regression model:

$$ f(\mathbf{x}) = \phi(\mathbf{x})^T \mathbf{w}$$ $$and$$ $$y = f(\mathbf{x}) + \epsilon$$ where:

  • $\mathbf{x}$ denotes the input vector for an input

  • $\phi(X)$ denotes some basis function applied on the input space

  • $\mathbf{w}$ is the weight vector

  • (other symbols defined as usual).

However non-linearity of inputs doesn't affect the linearity of the model itself. Kindly refer to the derivation in the weight-space view section of http://www.gaussianprocess.org/gpml/chapters/RW2.pdf for more information.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.