In what ways is Gaussian Process Regression both parametric and non-parametric?

Gaussian Process Regression is considered a "non-parametric" model. However, the term "non-parametric" is often used imprecisely to mean different things, leading to questions about what is non-parametric. This confusion is highlighted in the "non-parametric statistics" Wikipedia article, which discusses two interpretations of "non-parametric":

1. Techniques that do not rely on data belonging to any particular parametric family of probability distributions. These may consist of distribution-free methods where the data need not be drawn from a "parametric family" of probability distributions or consist of statistics defined to be a function on a sample, without dependency on a parameter.
2. Techniques that do not assume that the structure of a model is fixed. Typically, the model grows in size to accommodate the complexity of the data. In these techniques, individual variables are typically assumed to belong to parametric distributions, and assumptions about the types of associations among variables are also made. For example, non-parametric regression, which is modeling whereby the structure of the relationship between variables is treated non-parametrically, but where nevertheless there may be parametric assumptions about the distribution of model residuals.

Question:

In light of this, in what ways is Gaussian Process Regression parametric and in what ways is it non-parametric?

While Why are Gaussian process models called non-parametric? is a similar question, the question and accepted answers only (imprecisely) address how GPs might be considered non-parametric under one definition of non-parametric, rather than the different definitions above for (non)parametric models.

An ideal answer would be precise enough to discuss at least the following:

• Assumptions about kernels, covariance structures, and functions.
• Growth of parameters with N
• Priors over functions rather than specific parameters

Disclaimer: My background is in spatial statistics, and I'd expect my answer to contain some artifacts of the particulars of the way that Gaussian process models are used within that field. Nonparametrics is also a huge field that I am not an expert in, but I can try to relay some of the basics in a way that is (hopefully) mostly correct.

The math surrounding nonparametrics in general, and GPs in particular, is very interesting and I'd encourage you to consult a text like Giné and Nickl if you're interested in a more rigorous treatment.

Regression in General

In very general terms, a statistical model is a family of probability distributions $$\{P_\theta : \theta \in \Theta \}$$, where each $$P_\theta$$ is a candidate distribution for having generated the random observations of interest. These distributions are parameterized by elements of some parameter space $$\Theta$$.

The precise distinction between a non-parametric and parametric model is in the dimensionality of $$\Theta$$.

With regression specifically, we have paired observations $$(Y_i,X_i)$$, and we assume a functional relationship of the form $$Y_i = f(x_i)$$, generally with the addition of some $$\epsilon_i$$ to represent some kind of independent 'noise' associated with the measurement of $$Y_i$$.

Parametric Regression

In a parametric regression, the parameter space is assumed to be finite-dimensional - that is, the number of parameters associated with the model is fixed a-priori. The vanilla regression model looks something like this:

$$Y_i = \sum_{j=1}^p x_{ij} \beta_j + \epsilon_i \quad ; \quad \epsilon_i \sim \mathcal{N}(0, \sigma^2)$$

Here the distribution of $$Y_i$$ is parameterized by $$\theta = (\beta, \sigma^2)$$, consisting of the vector of regression coefficients and the residual standard deviation. If we assume real-valued coefficients, these parameters live in the $$(p+1)$$-dimensional space $$\Theta = \mathbb{R}^p \times (0, \infty)$$.

Nonparametric Regression

In nonparametric regression we keep the assumption that $$Y_i = f(x_i)$$, but not necessarily much else. Some models keep the noise component we saw above, some don't, but the general idea is that we want to relax some assumptions surrounding $$f$$ and solve the more general problem of approximating the function itself.

Since $$f$$ is an element of some function space, and function spaces are infinite-dimensional, we're stuck with the problem of describing an infinite-dimensional object without placing a restriction on how many parameters we can use.

Growth of parameters with N

In practice, this means that number of parameters is a function of N , but that's a limitation that arises from having a finite amount of data. (note that the number of parameters in a nonparametric model is not necessarily a monotone-increasing function of $$N$$; it all depends on the specific model).

In the infinite-sample case, a nonparametric model can absolutely have infinitely many parameters, while a parametric model can not. The object being approximated in a parametric model is a finite-dimensional vector, and its position in $$\mathbb{R}^{p+1}$$ is completely described by a fixed number of components, and all we can do is move their values around.

This is not the case for a nonparametric model, since we threw out that restrictive assumption - the number of parameters required to represent an arbitrary function can be anywhere between $$1$$ and $$\infty$$, and lots of other things will vary depending on the particulars of the model.

Gaussian Process Regression

In a GP model, we assume that our observations $$Y_i$$ arise as a result of "observing" a realization of an infinite-dimensional Gaussian process. These are extremely general mathematical objects, but for modeling purposes they're generally viewed through the lens of the Kolmogorov extension theorem and defined in terms of the Gaussian distribution of their fidis. Again, I'd recommend Giné and Nickl for an extremely comprehensive breakdown of the mathematical foundations.

So, as above, can look at the GP as a "random function" parameterized by two functions that describe its first-order and second-order behavior: the mean function, and covariance function respectively. For a GP indexed over some domain consisting of elements $$x$$, via severe abuse of notation, we can write this as

$$f(x) \sim \mathcal{GP}(\mu(x), C(x,x'))$$

In practice (at least in spatial statistics), we often center our observations so that $$\mu(x) = 0$$ identically for all $$x$$, and focus our attention on the covariance structure.

Also in practice, most GP models are actually semi-parametric (the parameter space has both finite-dimensional and infinite-dimensional components.) For instance, in spatial statistics we're often interested on performing inference about associations between measurements $$(Y_i, x_i)$$ observed at locations $$s \in \mathbb{R}^2$$. One typical model for this is the hierarchical GP model (see Banerjee et al or another spatial stats book for further details)

$$Y_i = \sum_{j=1}^p \beta_ix_{ij} + w(s_i) + \epsilon_i$$ $$w(s_i) \sim \mathcal{GP}(\mu(s), C(s,s'))$$

Here we have a parametric component in the form of the regression, while the GP is essentially being used to control for the error introduced by spatial correlation.

Kernels and covariance structures.

In my experience, at least in the context of GPs, kernel is ML-speak for covariance function. In the broader context of nonparametrics, kernel has a more general definition.

A valid covariance function fits the typical definition of a kernel function - in particular, the covariance function must be PSD for any realization to be a valid covariance matrix, so Mercer's theorem applies and we can do the kernel trick if we fix a basis. The Gram Matrix is then equivalent to a realized covariance matrix (see Rasmussen and Williams).

The kernel/covariance structure then, is essentially how we describe the second-order behavior of the GP. The big question in most GP modeling scenarios has to do with different stationarity assumptions on the GP. In general, this can either be constructed based on domain knowledge, or estimated in some way (e.g. via DNN as in here).

Why are GPs sometimes called parametric and sometimes not?

GPs are certainly not parametric models. While they are parameterized by functions, this results in an infinite-dimensional parameter space. They are, in a vacuum, non-parametric models, but in my experience they're often used as building blocks for structured semi-parametric models.

• +1 Thank you for the nice overview. Welcome to CV!
– whuber
Commented Apr 19 at 21:47