I am trying to understand intuitively how a kernel works in a Gaussian process. I know that GP are distributions over functions, in short you have the model $y = f(x)+\epsilon$ and the $f(x)$ follows a Gaussian process


After deriving the conditional distribution we arrive at the predictive equations (with noise):

$\hat{f}=K(x,x^{\prime})(K(x,x)+\sigma^{2}I)^{-1}y$ for the mean


$cov(f)=K(x^{\prime},x^{\prime})-K(x^{\prime},x)(K(x,x)+\sigma^{2}I)^{-1}K(x,x^{\prime})$ for the covariance.

The covariance kernel is the SE (Squared Exponential):

$k_{f}(x_{i},x_{j}) = \sigma^{2}exp(-\frac{1}{2\ell^{2}}\sum_{j=1}^{q}(x_{i,j}-x_{k,j})^{2})$

I know that a kernel, when we have a vector with inputs $x$ and size $(n)$, has dimensions $(n\times n)$. But how it works in multidimensional environment? For example, the covariance above $cov(f)$ with one dimensional inputs $(n)$ and for a single test point $(p = 1)$ will have the form:

$(p \times p) - (p \times n)(n \times n)(n \times p)$ which gives a scalar (the variance).

I can't comprehend how the above equation will work when we accept as input a matrix $(n \times d)$ size instead.

  • $\begingroup$ You might find this question helpful. Also the Kernel cookbook is a great resource for more standard kernels. $\endgroup$
    – combo
    Aug 22 '17 at 15:48
  • $\begingroup$ What is $d$? (The dimension of what?) Also, looks like the question is asking about some issue with multidimensional something rather than "understand intuitively how a kernel works"? $\endgroup$ Aug 23 '17 at 5:07
  • $\begingroup$ I have been clear on that. "We accept as input a matrix with dimensions $(n x d)$, so is the dimensions of matrix $x$. Now the question asks how the kernel works in a multi-dimension, in which case you are right. I do not state that explicitly but rather I have put forward an example, as the calculation of covariance function depends on the kernel. So, in that example, if we have a multidimensional input $x$, how do we calculate the covariance function? Thanks $\endgroup$
    – Jespar
    Aug 23 '17 at 5:36
  • $\begingroup$ Scrap that, I just realised that I also state "how the kernel works in a multidimensional environment". And then I provide an example. $\endgroup$
    – Jespar
    Aug 23 '17 at 5:41
  • $\begingroup$ Please read the question carefully, "one dimensional inputs (n)". (p) is the size of the test point. I do state that. So you have as input a one dimensional vector with size n and a scalar test point. I do not know how to be more clear on that. Yes, the question asks "how it works in a multi dimensional environment". $\endgroup$
    – Jespar
    Aug 23 '17 at 5:55

Notation / setting

We are considering a GP regression model: \begin{equation} y_i = f(x_i) + \epsilon_i \end{equation} where $y_i\in \mathbb{R}$,$x_i \in \mathbb{R}^d$, $f$ a Gaussian process (whose realizations are functions $f:\mathbb{R}^d\rightarrow \mathbb{R}$), \begin{equation} f \sim \mathrm{GP}(m(x_i), \kappa(x_i,x_j)). \end{equation} $n$ datapoints $(y_1,x_1), (y_2,x_2), (y_3,x_3),\ldots, (y_n,x_n)$ are given. (I use $\kappa$ to distinguish the function from the matrices $K(\cdot,\cdot)$ that contain values of $\kappa$ evaluated at certain points. The question denotes both by $K$)

How to handle $d$-dimensional inputs

The question covers computing the posterior predictive distribution for a test point (or $p$ test points) in the case $d=1$ and asks how to extend to the general $d=2,3,\ldots$.

Answer: nothing changes and the formulas from the one-dimensional case work as well in this case. Note that $m$ is then a function from $\mathbb{R}^d$ to $\mathbb{R}$ and $\kappa$ a function from $\mathbb{R}^d \times \mathbb{R}^d$ to $\mathbb{R}$.

So, for example the matrix denoted by $K(x,x)$ in the question is a $n\times n$ matrix for which $K(x,x)_{i,j} = \kappa(x_i, x_j)$ ($x_i$ and $x_j$ are $d$-dimensional but since $\kappa$ maps two $d$-dimensional vectors to a scalar, $\kappa(x_i, x_j)$ is a scalar. Similarly for $K(x,x')$ and $K(x',x')$ where $x'$ are the test points.

Thus, the dimensions of the matrices in the predictive covariance equation are $(p\times p) - (p \times n)\,(n \times n)\,(n \times p)$ independent of whether the elements of the matrices are obtained by evaluating a function $\kappa(\cdot,\cdot)$ whose arguments are $1$-dimensional of a function $\kappa(\cdot, \cdot)$ whose arguments are $d$-dimensional. In fact, the inputs could even be in some space other than $\mathbb{R}^d$ (such as if we have a categorical predictor) as long as a positive-definite covariance function can be defined.

An extra remark about the SE kernel appearing in the question

The question mentions the SE kernel \begin{equation} k_{f}(x_{i},x_{j}) = \sigma^{2}\exp\!\Big(-\frac{1}{2\ell^{2}}\sum_{j=1}^{q}(x_{i,j}-x_{k,j})^{2}\Big) \end{equation} Note that this is already a function from $\mathbb{R}^q \times \mathbb{R}^q$ to $\mathbb{R}$ (with scalar inputs there would be no "$x_{i,j}$ and $x_{k,j}$" for different values of $j$. And $q$ should be $d$ if $d$ is the dimension of inputs.

Optionally, the length scale $\ell$ could be made different for each input dimension as $\ell_{j}$, such that the term $\frac{1}{2\ell_{j}^{2}}$ is instead placed inside the summation

  • 1
    $\begingroup$ Especially for the last part, about the kernel. Very helpful indeed! I should delete my previous 'answer' $\endgroup$
    – Jespar
    Aug 23 '17 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.