I'm trying to understand how a Gaussian Process with a squared exponential covariance function can be obtained from Bayesian Linear Regression with a Gaussian prior $N(0,\sigma_p^2 I)$ on the parameters and an infinite number of basis functions. I'm following the proof in chapter four of Gaussian Processes for Machine Learning.

Let $\phi_c(x)=\exp(-\frac{(x-c)^2}{2\ell^2})$, where $c$ is the center of the basis function. Then Bayesian Linear Regression is a Gaussian Process with mean function $\mu(x)= 0$ and covariance function $k(x_p,x_q)=\sigma_p^2\sum_{c=1}^N\phi_c(x_p)\phi_c(x_q)$. Here's the part of the proof where I'm confused because it feels a little hand-wavy.

Now, allowing an infinite number of basis functions centered everywhere on an interval (and scaling down the variance of the prior on the weights with the number of basis functions) we obtain the limit $$ \lim_{N\to\infty}\frac{\sigma^2_p}{N}\sum_{c=1}^N\phi_c(x_p)\phi_c(x_q)=\sigma^2_p\int_{c_{\text{min}}}^{c_{\text{max}}}\phi_c(x_p)\phi_c(x_q)dc. $$ Plugging in the Gaussian-shaped basis functions and letting the integration limits go to infinity we obtain $$ \begin{align*} k(x_p, x_q) &= \sigma^2_p\int_{-\infty}^\infty\exp\Big(-\frac{(x_p - c)^2}{2\ell^2}\Big)\exp\Big(-\frac{(x_q - c)^2}{2\ell^2}\Big)dc\\ &=\sqrt{\pi}\ell\sigma^2_p\exp\Big(-\frac{(x_p - x_q)^2}{2(\sqrt{2}\ell)^2}\Big) \end{align*} $$ which we recognize as a squared exponential covariance function with a $\sqrt{2}$ times longer length-scale.

I'm hoping someone could add some further explanation to this proof. Why must the covariance function be scaled by a factor of $1/N?$ Also, it's unclear to me why we can simply change the limits of integration from $c_{\text{min}}$ and $c_{\text{max}}$ to $\pm\infty$. Maybe this is because I'm not seeing how the limit of the covariance function corresponds to having an infinite number of basis functions centered densely in $x$-space.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.