1
$\begingroup$

The other day a coworker was claiming that Gaussian Process Regression with a squared exponential kernel (from now on, GPR) for a data set $D=\{\mathbf{x}_i,y_i\}_{i=1}^N$ could be interpreted as just summing $N$ Gaussian basis functions centered at the points $\mathbf{x}_i$, each one multiplied by $y_i$, which sounds like (but it's not exactly the same as) approximation by Radial Basis Functions (RBF) with a squared exponential kernel. I think my coworker is wrong, and I would like to get your opinion on this.

Basically, (my understanding of) his proposal is to predict $y$ at $\mathbf{x}$ as

$y(\mathbf{x})\approx\hat{y}(\mathbf{x})=\sum_{i=1}^N y_i \exp\left(-\frac{||\mathbf{x}-\mathbf{x}_i||^2}{2\sigma}\right)\tag{1}\label{1}$

where presumably $\sigma$ would be set to the value which minimizes the RMSE on the training set (or better, the value which minimizes cross-validation error). I'm not even sure this would work in practice, because the value of $\sigma$ which minimizes the training set RMSE is zero (a Dirac delta at each point). Maybe the cross-validation criterion wouldn't suffer by the same problem.

This is not exactly the same as approximation by RBFs, because in that case the weights are not equal to the response values $y_i$, but are additional parameters to be determined. For RBFs, the prediction equation would be

$\hat{y}(\mathbf{x})=\sum_{i=1}^N w_i \exp\left(-\frac{||\mathbf{x}-\mathbf{x}_i||^2}{2\sigma}\right) \tag{2}\label{2}$

Now, this formula may look like the usual formula for the posterior mean in Gaussian Process regression

$\mu(\mathbf{x}|D)= \mathbf{k}^T(\mathbf{x}) (K+\sigma_n^2I)^{-1}\mathbf{y}\tag{3}\label{3}$

because

$\mathbf{k}^T(\mathbf{x})=\sigma_s^2\left(\exp\left(-\frac{||\mathbf{x}-\mathbf{x}_1||^2}{2\sigma}\right), \exp\left(-\frac{||\mathbf{x}-\mathbf{x}_2||^2}{2\sigma}\right),\dots,\exp\left(-\frac{||\mathbf{x}-\mathbf{x}_N||^2}{2\sigma}\right)\right)$

where to simplify the connection with RBFs I chose a constant correlation length in all dimensions, even if this is not the usual choice in GPR*. Thus $\eqref{2}$ and $\eqref{3}$ may look quite similar, even if I'm not sure the weights would be the same in practice. But definitely the weights are not the response values as in $\eqref{1}$. Am I right?

* Note also that in GPR usually the symbol $\sigma$ is not used for the correlation length, in order to avoid confusion with $\sigma_s$ (the signal variance) and $\sigma_n$ (the noise variance in the nugget term), but again, I tried to use a similar notation for GPR and RBF.

$\endgroup$
  • $\begingroup$ I thought about the same thing before. I guess you are right. The weights are more complicated than just $\pm1$s. The vector of $\pm1$s are multiplied by $(K+\sigma_n^2I)^{-1}$. It is still interpolation using RBF basis and weights are $(K+\sigma_n^2I)^{-1} \mathbf{y}$. $\endgroup$ – Seeda Jan 31 '17 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.