I am new to this site as well as Machine learning, so kindly bear with me.

I have been trying to understand Gaussian process and their implementation.


1) Let's say that the $\vec{x}$ $\in R^{n}$ and $f(\vec{x})$ $\in R$

2) I have m data points of ($\vec{x}$,$f(\vec{x})$)

Things I have understood:

1) Gaussian process gives a distribution over functions that might be possible given data ($\vec{x}_{p}$,$f(\vec{x}_{p})$), p = 1....m

So the squared exponential kernel defined by:

$k(x_{i},x_{k}) = \sigma^{2}exp\left(\frac{-1}{2}\Sigma_{j=1}^{q}\left(\frac{x_{i,j}-x_{k,j}}{l}\right)^2\right)$

Things I have not clearly understood:

1) What is $x_{i},x_{k}$ and what are their dimensions?

2) What is covariance matrix and what is its dimension?

3) What is the value of q, is it 'n' ?

4) What is the size of the kernel? is it a matrix of size m x m or n x n?

As you might have guessed I am pretty new to all this. If you could give a small three data points and two dimensions input to demonstarte the sizes of kernel and co variance matrix.

Thank you for your time.

  • 3
    $\begingroup$ I would recommend reading a high-quality text on the topic, such as Rasmussen and Williams' Gaussian Processes for Machine Learning. gaussianprocess.org/gpml $\endgroup$ – Sycorax Sep 25 '19 at 11:46

1) $x_i$ can be any dimension, this is simply the dimension of your data. $x_k$ is just a different training example, so will have the same dimension as $x_i$. In the example you have it looks like the dimension of the data is $q$. This is exactly analogous to the regression case where you can have regressors that have any dimension. In the simplest case the dimension is 1.

2) the size (# of rows and # of columns) of the covariance matrix is equal to the number of data points. The covariance function specifies the covariance between any two test points. For example, to get the entry (i,k) of the covariance matrix, you plug $x_i$ and $x_k$ into the covariance function.

3) The value of q is equal to the dimensions of $x_i$. This sum is simply a sum over the dimensions of $x_i,x_k$

4) The output of the kernel function should be dimension 1 (see answer to 2). That means that the full covariance matrix is $m\times m$

  • $\begingroup$ Thanks a lot. I appreciate your time and effort. $\endgroup$ – Tyrion Sep 25 '19 at 9:38
  • 1
    $\begingroup$ Happy to help. If you haven't already found it, this is a great resource gaussianprocess.org/gpml/chapters $\endgroup$ – user3235916 Sep 25 '19 at 9:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.