Why Gaussian Process Regression (GPR) is non-parametric?

Question

Given that Gaussian Process (GP) regression relies on a kernel with specific hyperparameters that control the relationships and smoothness between points, can GP regression truly be considered a non-parametric technique? I have the following points.

1. A GP's behavior is largely determined by the kernel's parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., AR models), as the amount of data grows, GPs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure. Furthermore, the ability to draw multiple functions from the posterior distribution (as in parametric models with sampling or bootstrap approaches) does not imply a non-parametric model.

3. Though each GP realization represents a potential function, they all adhere to the underlying rule established by the kernel, which effectively governs the ways samples can fit the observed data. Consequently, labeling GPs as non-parametric merely because the function form is not entirely predictable may be inaccurate, as the kernel parameterizes and constrains GP flexibility.

In summary, should GP regression not be considered a parametric method due to the kernel's role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric kernel estimator which when used with the Bayesian framework, produces posterior predictions.

Answer 1

The points you raised apply to K Nearest Neighbors Regression too.

Let me start by rephrasing what you wrote in terms of K-nearest neighbors regression:

"""

Given that K nearest neighbor (KNN) regression relies on a distance metric with specific hyperparameters [Number of neighbors $K$, Parameter $p$ and dimension wieghts $w_p$ in $\sum_{d=1}^D w_d (|x_{1,d}-x_{2,d}|^p)^{\frac{1}{p}}$] that control the dissimilarities between points, can KNN regression truly be considered a non-parametric technique? I have the following points.

1. A KNN regressor's behavior is largely determined by the distance's parametric structure, which restricts the functional form to be consistent with certain assumptions (e.g., smoothness, periodicity). This makes it a deterministic process governed by hyperparameters, not an inherently flexible, parameter-free model.

2. Like any parametric technique (e.g., DFT or AR models), as the amount of data grows, KNNs allow for more precise estimates of hyperparameters that better fit the data. This improvement with data could be considered similar to reducing sub-optimality in parameter estimation rather than reflecting an open, parameter-free structure.

In summary, should KNN regression not be considered a parametric method due to the distance metric/$K$'s role as a definitive, parameterized structure that guides model behavior? In my opinion, it is indeed a parametric estimator.

"""

[Note: I have excluded Point 3 and other text about the fact that a GP is a probabilistic model, which is not relevant.]

So if GPs are parametric, so is KNN regression. Are they indeed both parametric?

---

How to tell the difference.

In my view, none of what you have mentioned has to do with parametric vs nonparametric. It might be confusing that "nonparametric" methods still have parameters, but they all do.

So how do we tell the difference? I like to use the following demarcation. A method is nonparametric if, after estimating the parameters, we still need the data to make predictions. Otherwise, if having the parameters means we can throw away the data and still make predictions, it is parametric.

Some examples. If I throw away the data, can I still make predictions at point $\mathbf{x}$?

1. Linear regression: yes, I just calculate $\beta^\top\mathbf{x}$. It must be parametric!
2. Neural network: yes, I just push $\mathbf{x}$ through my network. It must be parametric!
3. K nearest neighbors: no; just knowing $K$ and the distance metric is not enough to make any predictions: I need to know the average at the $K$ closest points to $\mathbf{x}$ under my metric. It must be nonparametric.
4. Gaussian processes: knowing a kernel function's parameters are not enough to make any predictions: I need to know the correlation between all training points and $\mathbf{x}$ as well as the intra-training correlations. Hence it is nonparametric.

Answer 2

Generally speaking, with the way people use the term nonparametric to decribe a model, if you had more data points, there would be more parameters, generally without an upper limit. It doesn't mean "has no parameters" nor is it required to be infinite-parametric with a finite sample size.

To take a different example, consider say spline models. Even though with some given sample you could give a list of parameters and their estimates (that is, after all, how you calculate the smooth fit), it is still nonparametric in this sense of the word.

Similarly nonparametric distributional models for continuous data can be based on things like the ecdf itself or on a histogram or on a KDE. In particular note that with the KDE, even though for a given kernel family (say Gaussian), the kernel itself is "parametric" in the sense that it has a single parameter, the density estimate itself is not fixed-parametric.

In each such case, the number of parameters that describes the function is not fixed and will generally grow with sample size (not always linearly).