A commonly used kernel in Gaussian processes is the RBF kernel:
$$ \kappa(x,x') = \exp\left(-\frac{|| x-x'||^2}{2\sigma^2}\right) $$
In the context of a Gaussian process, a kernel is used to construct a covariance matrix
$$ \Sigma_{ij} = \kappa(x_i,x_j) $$
for $x_i$ and $x_j$ in a given space.
Covariance matrices have to be positive definite. However, I ran into the problem when implementing this that they are not always:
xa = xb = Subdivide[-5, 5, 100];
m = Outer[Exp[-0.5 Norm[# - #2]^2] &, xa, xb];
PositiveDefiniteMatrixQ[m]
(* False *)
In this Mathematica code I first took 101 points uniformly from the range $[-5,5]$ and then evaluated $\Sigma_{ij}$ with these points for all $i, j$. I used $\sigma = 1$ in the kernel. The corresponding matrix $\Sigma$ is not positive definite.
I read here that such a covariance matrix is not guaranteed to be positive definite. I also read here that it can be fixed by adding a small constant to the diagonal of the covariance matrix, which seems to be correct.
xa = xb = Subdivide[-5, 5, 100];
m = Outer[Exp[-0.5 Norm[# - #2]^2] &, xa, xb];
m += DiagonalMatrix[ConstantArray[0.0001, 101]];
PositiveDefiniteMatrixQ[m]
(* True *)
My questions are the following:
- How is the covariance matrix usually created in practice? It seems to be a very widely used kernel for Gaussian processes, and there should be a standard way of creating the covariance matrix from this kernel in such a way that it is positive definite.
- Why does the trick of adding a small constant to the diagonal of the covariance matrix work?
- The second post I linked to says that the problem is that since the RBF kernel is very smooth there is a singularity for numbers that are very close to each other. Where does this singularity come from? I know that the numbers will be similar to one another if they are close in input space, but I don't see any singularity.