Is Kernel Regression similar to Gaussian Process Regression? I've used Nadaraya-Watson Kernel regression before to smooth data.  Recently I have run into Gaussian process regression.
Prima facie, they don't seem to be related.  But I am wondering if there perhaps is a deeper connection I am not aware of.  Is Nadaraya-Watson kernel regression a special case of GPR?
 A: Yes, there is a connection, depending on the GP covariance function and the kernel of the smoother. It's discussed in chapter 2 (section 2.6) of Gaussian Processes for Machine Learning. Note that even a simple covariance function, such as the squared exponential, results in complex equivalent kernels due to the spectral properties of the function. 
Other things to note are:


*

*in the multivariate setting, the N-WKR boils down to univariate regression in each dimension (see this answer), whereas GPs can model the full multivariate covariance. 

*there is no equivalent to the GP mean function

*the kernel in N-WKR needn't be a valid GP covariance function, and there may not be an equivalent covariance function for every kernel

*there is no obvious equivalent for e.g. periodic covariance functions as a kernel smoother

*in GPs you are free to combine covariance functions (e.g. through multiplication or addition), see e.g. the kernel cookbook
A: There is a connection in that Gaussian Process Modeling is a kernel technique, meaning that GPMs use a kernel function to describe a multivariate Gaussian covariance among observed data points, and regression is used to find the kernel parameters (hyperparameters) that best describe the observed data.  Gaussian Process Modeling can extrapolate from observed data to produce an interpolating mean function (with associated uncertainty dictated by the kernel function) for any point in the space.
Below are some resources on GPM that describe in detail what types of kernel functions are typically employed as well as the approaches used to estimate kernel hyperparameters:
http://www.gaussianprocess.org/gpml/
http://www.eurandom.tue.nl/events/workshops/2010/YESIV/Prog-Abstr_files/Ghahramani-lecture2.pdf
