# Is Gaussian Process Regression a linear model?

I had a discussion today with someone saying that Gaussian Processes are linear models. I don't see in which sense this may be correct. To be clear, here the definition of a linear model is the usual one, i.e., a model which is linear in the parameters. Thus,

$y=\beta_0+\beta_1x+\beta_2\sin(x)+\epsilon$

and

$y=\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x}+\epsilon$

are linear, and

$y=\beta_0+\beta_1x+\beta_2\exp({\beta_3x})+\epsilon$

is not.

For simplicity, let's consider a Squared Exponential covariance function, and assume that the correlation length, the signal variance and the noise variance are known. Given a design matrix $X$ and corresponding response vector $\mathbf{y}$, the GP prediction at a new prediction point $\mathbf{x}^*$ is

$$\hat{y}(\mathbf{x}^*)=\mathbf{k}_*^T(K+\sigma I)^{-1}\mathbf{y}$$

Now, this estimator is clearly a nonlinear function of $X$ and a linear function of $\mathbf{y}$. The other person insisted that $\mathbf{y}$ is the parameter vector of this model, and thus the model is linear. I don't think this makes any sense: it would mean that the number of parameters of the model depends on the sample size. I think we can at most say that the estimator is a linear function of $\mathbf{y}$, but surely not the statistical model underlying Gaussian Process Regression. Do you agree?

• Have you noticed that the OLS estimator $(X^\prime X)^{-}X^\prime y$ is highly nonlinear in $X$, too? – whuber Oct 31 '17 at 19:43
• @whuber sure, and it's also linear in $\mathbf{y}$, like the GPR estimator. However, I was talking about linearity of the model, not the estimator. I guess the other person confounded the two things. – DeltaIV Oct 31 '17 at 21:58
• Interestingly, a GP regression model is equivalent to (linear) ridge regression if you use a linear kernel. – Kevin Yang Nov 1 '17 at 1:13
• @KevinYang true, but a linear kernel is a degenerate kernel, in that its eigenexpansion is finite (and thus the covariance function is only semidefinite positive, instead than definite positive). Let's consider just strictly positive definite covariance functions: do you think GP regression is a linear model or not? If it is, and you can show me which are the model parameters I would accept it as an answer. However, if the parameters depend on the data set, I don't think we can talk of a linear model...ideally the model is defined independently of the data set. – DeltaIV Nov 1 '17 at 11:25