5
$\begingroup$

I had a discussion today with someone saying that Gaussian Processes are linear models. I don't see in which sense this may be correct. To be clear, here the definition of a linear model is the usual one, i.e., a model which is linear in the parameters. Thus,

$y=\beta_0+\beta_1x+\beta_2\sin(x)+\epsilon$

and

$y=\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x}+\epsilon$

are linear, and

$y=\beta_0+\beta_1x+\beta_2\exp({\beta_3x})+\epsilon$

is not.

For simplicity, let's consider a Squared Exponential covariance function, and assume that the correlation length, the signal variance and the noise variance are known. Given a design matrix $X$ and corresponding response vector $\mathbf{y}$, the GP prediction at a new prediction point $\mathbf{x}^*$ is

$$\hat{y}(\mathbf{x}^*)=\mathbf{k}_*^T(K+\sigma I)^{-1}\mathbf{y}$$

Now, this estimator is clearly a nonlinear function of $X$ and a linear function of $\mathbf{y}$. The other person insisted that $\mathbf{y}$ is the parameter vector of this model, and thus the model is linear. I don't think this makes any sense: it would mean that the number of parameters of the model depends on the sample size. I think we can at most say that the estimator is a linear function of $\mathbf{y}$, but surely not the statistical model underlying Gaussian Process Regression. Do you agree?

$\endgroup$
  • 2
    $\begingroup$ Have you noticed that the OLS estimator $(X^\prime X)^{-}X^\prime y$ is highly nonlinear in $X$, too? $\endgroup$ – whuber Oct 31 '17 at 19:43
  • 1
    $\begingroup$ @whuber sure, and it's also linear in $\mathbf{y}$, like the GPR estimator. However, I was talking about linearity of the model, not the estimator. I guess the other person confounded the two things. $\endgroup$ – DeltaIV Oct 31 '17 at 21:58
  • 1
    $\begingroup$ Interestingly, a GP regression model is equivalent to (linear) ridge regression if you use a linear kernel. $\endgroup$ – Kevin Yang Nov 1 '17 at 1:13
  • $\begingroup$ @KevinYang true, but a linear kernel is a degenerate kernel, in that its eigenexpansion is finite (and thus the covariance function is only semidefinite positive, instead than definite positive). Let's consider just strictly positive definite covariance functions: do you think GP regression is a linear model or not? If it is, and you can show me which are the model parameters I would accept it as an answer. However, if the parameters depend on the data set, I don't think we can talk of a linear model...ideally the model is defined independently of the data set. $\endgroup$ – DeltaIV Nov 1 '17 at 11:25
3
$\begingroup$

I think the technically correct term to use here is that GP regression is a linear smoother, i.e. its predictions are a linearly weighted combination of past observed outputs. This does not make the model as such linear. For that to be true, the predictions must be a linear function of the inputs. This is only the case with GPs if you use a linear covariance function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.