Is Gaussian Process Regression a linear model? I had a discussion today with someone saying that Gaussian Processes are linear models. I don't see in which sense this may be correct. To be clear, here the definition of a linear model  is the usual one, i.e., a model which is linear in the parameters. Thus,
$y=\beta_0+\beta_1x+\beta_2\sin(x)+\epsilon$
and
$y=\beta_0+\boldsymbol{\beta}^T\cdot\mathbf{x}+\epsilon$
are linear, and
$y=\beta_0+\beta_1x+\beta_2\exp({\beta_3x})+\epsilon$
is not.
For simplicity, let's consider a Squared Exponential covariance function, and assume that the correlation length, the signal variance and the noise variance are known. Given a design matrix $X$ and corresponding response vector $\mathbf{y}$, the GP prediction at a new prediction point $\mathbf{x}^*$ is
$$\hat{y}(\mathbf{x}^*)=\mathbf{k}_*^T(K+\sigma I)^{-1}\mathbf{y}$$
Now, this estimator is clearly a nonlinear function of $X$ and a linear function of $\mathbf{y}$. The other person insisted that $\mathbf{y}$ is the parameter vector of this model, and thus the model is linear. I don't think this makes any sense: it would mean that the number of parameters of the model depends on the sample size. I think we can at most say that the estimator is a linear function 
 of $\mathbf{y}$, but surely not the statistical model underlying Gaussian Process Regression. Do you agree?
 A: I think the technically correct term to use here is that GP regression is a linear smoother, i.e. its predictions are a linearly weighted combination of past observed outputs.  This does not make the model as such linear. For that to be true, the predictions must be a linear function of the inputs.  This is only the case with GPs if you use a linear covariance function.
A: The statistical definition of a model being linear is that the model must be linear in its parameters.
Gaussian Process Regression can be defined by using either the function-space view or the weight-space view to reach the formula for the posterior mean and posterior variance.
If we see the weight-space view, we can clearly see that Gaussian Process Regression is indeed a linear model with non linear functions of the inputs. The model is defined as a bayesian linear regression model:
$$ f(\mathbf{x}) = \phi(\mathbf{x})^T \mathbf{w}$$
$$and$$
$$y = f(\mathbf{x}) + \epsilon$$
where:

*

*$\mathbf{x}$ denotes the input vector for an input


*$\phi(X)$ denotes some basis function applied on the input space


*$\mathbf{w}$ is the weight vector


*(other symbols defined as usual).
However non-linearity of inputs doesn't affect the linearity of the model itself. Kindly refer to the derivation in the weight-space view section of http://www.gaussianprocess.org/gpml/chapters/RW2.pdf for more information.
