12
$\begingroup$

I've used Nadaraya-Watson Kernel regression before to smooth data. Recently I have run into Gaussian process regression.

Prima facie, they don't seem to be related. But I am wondering if there perhaps is a deeper connection I am not aware of. Is Nadaraya-Watson kernel regression a special case of GPR?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

Yes, there is a connection, depending on the GP covariance function and the kernel of the smoother. It's discussed in chapter 2 (section 2.6) of Gaussian Processes for Machine Learning. Note that even a simple covariance function, such as the squared exponential, results in complex equivalent kernels due to the spectral properties of the function.

Other things to note are:

  • in the multivariate setting, the N-WKR boils down to univariate regression in each dimension (see this answer), whereas GPs can model the full multivariate covariance.
  • there is no equivalent to the GP mean function
  • the kernel in N-WKR needn't be a valid GP covariance function, and there may not be an equivalent covariance function for every kernel
  • there is no obvious equivalent for e.g. periodic covariance functions as a kernel smoother
  • in GPs you are free to combine covariance functions (e.g. through multiplication or addition), see e.g. the kernel cookbook
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ 1) Do you have a citation for the GP regression with multivariate response? 2) I would have interpreted the N-WKR as the equivalent to the mean of the GP regression. What the N-WKR lacks would be the variance in this interpretation, not the mean. 4) Why can't you simply use a periodic function as a kernel for N-WKR? N-WKR is a convolution of the kernel and the data - just convolute the data with a periodic function. 5) I am not sure I understand the additive kernels in GP yet. $\endgroup$
    – Make42
    Commented Mar 2, 2018 at 18:16
-1
$\begingroup$

There is a connection in that Gaussian Process Modeling is a kernel technique, meaning that GPMs use a kernel function to describe a multivariate Gaussian covariance among observed data points, and regression is used to find the kernel parameters (hyperparameters) that best describe the observed data. Gaussian Process Modeling can extrapolate from observed data to produce an interpolating mean function (with associated uncertainty dictated by the kernel function) for any point in the space.

Below are some resources on GPM that describe in detail what types of kernel functions are typically employed as well as the approaches used to estimate kernel hyperparameters:

http://www.gaussianprocess.org/gpml/

http://www.eurandom.tue.nl/events/workshops/2010/YESIV/Prog-Abstr_files/Ghahramani-lecture2.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.