What are the limitations of Gaussian process regression and gaussian response surface methodologies? In which scenarios other techniques might be preferable over Gaussian process regression?
1
-
$\begingroup$ Also see this great answer to a related question (not exactly a duplicate). $\endgroup$– STJCommented Feb 16, 2021 at 12:09
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
9
GPs assume a Gaussian uncertainty on the $y$-values. However, this may not be the type of uncertainty that you have. For example, let us assume the output values are strictly positive, or bounded between two values, then the Gaussian prior would be inappropriate (or used only as an approximation).
SVMs are somewhat similar as they are kernel-based regression models for which you can choose your loss function. However, they don't offer a probabilistic interpretation (which is a big no no if you are a die hard Bayesian for example).
Kernel methods versus random forests or neural nets have other trade-offs. A GP kernel allows us to specify a prior on our function space which can be extremely useful especially when we have little data. However, a poor choice of kernel which specifies misconceptions about the function space can make convergence slow. Specifying appropriate kernels beyond the most basic requires some mathematical understanding. On the other hand, random forests and neural nets are completely frequentist (in general) and so usually require more data in order to get decent predictive performance.
-
$\begingroup$ Thanks for this great answer. However, I've commonly heard that the Gaussian Process is a nonparametric model, which implies it doesn't make any assumptions regarding the underlying distribution of the data. Yet it seems like the GP is, as you said, making a Gaussian uncertainty assumption on the output values? $\endgroup$– Yu ChenCommented Jun 17, 2018 at 18:48
-
1$\begingroup$ @YuChen I believe your definition of non-parametric is slightly off. GPs are non-parametric if they use a non-degenerate kernel. In this case the number of eigen values or degrees of freedom grows with the size of the data. In practice you run into computational constraints and usually fall back on a sparse approximation which makes the GP parametric once more. In both cases the noise on the output is assumed as Gaussian (there are of course exceptions to this addressed in a few papers but generally speaking this is the case). $\endgroup$– j__Commented Jun 17, 2018 at 18:57
-
$\begingroup$ Apologies for being a bit slow, but could you let me know exactly what part of my definition of non-parametric is off? I understand what you are saying about non-degenerate kernels and sparse approximations- but it sounds like what you are saying is that a "pure" GP is non-parametric but implementation constraints usually require some assumptions and approximations that make it parametric in practice? $\endgroup$– Yu ChenCommented Jun 17, 2018 at 19:00
-
3$\begingroup$ Ok sure, imagine we have a classic GP with an RBF kernel. It is known that this kernel lab model the space of all function. However, when a noiseless point is observed now it can only model the space of functions which pass through that point. As data is generally noisy practitioners add a diagonal term to the covariance matrix which is iid Gaussian noise. If that term is added we don't fully believe the observation and we return to being able to model the space of all functions once again. Of course we can use noiseless GPs but they often have conditioning issues. One could also use $\endgroup$– j__Commented Jun 17, 2018 at 19:11
-
3$\begingroup$ ....Non-Gaussian noise but then the GP is no longer closed form. The definition of non-parametric is that no parametric form is assumed on the function. By using a non-degenerate kernel the GP can fit to any function in the limit of data. However the fact Gaussian noise is generally assumed just dictates what happens before the limit of data haha I hope that make sense!! $\endgroup$– j__Commented Jun 17, 2018 at 19:17