So in GP the correlation between random variables is encoded by kernel function. I have few question in this regard:

  • Is the learned hyperparameters of the kernel such as lengtscale directly relate to the correlation between the variables? In other words if i am at variable $x$ does $x+L$ & $x-L$ determine the interval of the neighbourhood that the variable x is correlated with? Here $L$ is the lengthscale of the kernel function such as RBF kernel.

The Gaussian RBF kernel, also known as the squared exponential or exponentiated quadratic kernel, is $$ k(x, y) = \exp\left( - \frac{\lVert x - y \rVert^2}{2 \ell^2} \right) ,$$ where $\ell$ is often called the lengthscale. Remember that for $f \sim \mathcal{GP}(0, k)$, the correlation between $f(x)$ and $f(y)$ is exactly $k(x, y)$. So with a Gaussian RBF kernel, any two points have positive correlation – but it goes to zero pretty quickly as you get farther and farther away.

  • When $x$ and $y$ are $\ell$ apart, the correlation is $\exp(- \frac{\ell^2}{2 \ell^2}) = \exp(-\frac12) \approx 0.61$ – a decent amount of correlation.
  • $2 \ell$ apart means correlation $\exp(-\frac{2^2}{2}) \approx 0.14$ – only slightly dependent.
  • $3 \ell$ apart means correlation $\exp(-\frac{3^2}2) \approx 0.01$ – barely dependent.
  • $4 \ell$ apart means correlation $\exp(-\frac{4^2}{2}) \approx 0.0003$ – essentially independent for almost all practical purposes.

Other kernels will have different behavior here and different meaning for the lengthscale. For instance, the exponential or Laplace kernel $\exp(-\lVert x - y \rVert / \ell)$ has smaller correlation $0.37$ at one lengthscale, but has much heavier tails: it still has correlation $0.02$ at 4 lengthscales, 55 times as much as the Gaussian kernel does.

| cite | improve this answer | |
  • $\begingroup$ Makes so much sense, So hypothetically if I want to quantify the neighborhood. What shall be the value for decent amount of correlation? The reason I asked this question is I want to determine if some random variable from a query falls under the neighborhood of some optimal point $x$ in GP. So what metric can I use to determine if it falls under the neighborhood of that point? $2L$ or is there a better alternative? $\endgroup$ – GENIVI-LEARNER Jan 19 at 22:08
  • $\begingroup$ Or there is no such thing as neighbourhood as you mentioned that merely choosing different kernel function will lead to different correlation value. In that case I shall be doomed. $\endgroup$ – GENIVI-LEARNER Jan 19 at 22:18
  • $\begingroup$ Also +1 for great intuitive answer! $\endgroup$ – GENIVI-LEARNER Jan 19 at 22:18
  • $\begingroup$ @GENIVI-LEARNER It depends entirely on what you want to do with this neighborhood. If you want, you could choose a kernel which is actually 0 after some distance; this would allow for such a definition, but would also change your model, maybe significantly. $\endgroup$ – Dougal Jan 20 at 3:27
  • $\begingroup$ yes i did realize that, neighborhood depends entirely on the kernel. I just want to classify the point and its a good approximate if it falls under the neighborhood of some optimal point. So I am trying to see what are my options. $\endgroup$ – GENIVI-LEARNER Jan 20 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.