Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/eigenfuncions associated with the integral operator $T_K$ of $K$. Now, consider for example $K(x,y) = e^{-c|x-y|}$, that is the exponential kernel. From Stochastic Finite Elements: A Spectral Approach written by Ghanem and Spanos (Stochastic Finite Elements: A Spectral Approach), we know an expression of the eigenvalues/eigenfunctions (see p.28/29). Consider an approximation of the kernel by $K_{N}(x,y) = \sum\limits_{i=1}^{N}\lambda_ie_i(x)e_i(y)$ and we know from Mercer's theorem that $K_{N}$ converges uniformly to $K(x,y)$ as $N\to+\infty$. Take the first eigenvalue/eigenfunction of $T_K$, that is $K_{1}(x,y) = \lambda_1e_1(x)e_1(y)$. From Ghanem's book, the expression of is $e_1(x) = \frac{cos(\omega x)}{\sqrt{a + \frac{sin(2\omega a)}{2\omega}}}$ where $\omega$ is the first solution of $c - \omega tan(\omega a)= 0 $.

Finally, take 3 points $x_0 = -1, x_1 = 0 ,x_2 = 1$ and form the Gram matrix $K_{ij} = \lambda_1e_1(x_i)e_1(x_j)$ and you want to us this Gram matrix for regression. One knows that the value of at an unobserved point $x_*$ is $f(x_*) = K_1(x_*, X)K_1(X,X)^{-1}Y$ where $X = (x_0, x_1, x_2)$. The issue is that $K_1(X,X)$ is not invertible as the first row is equal to the third row and so the determinant of $K_1(X,X)$ is equal to 0 but $K(X,X)$ is invertible. What am I missing ?


1 Answer 1


The problem is that in your derivation you omit the ``truncation noise'' which must be taken into account.

Mercer's theorem can be used to get a finite-rank approximation of a kernel. In terms of model, the Gaussian Process $Y(\mathbf{x})$ is approximated by $$ Y(\mathbf{x}) = \alpha_1 e_1(\mathbf{x}) + \alpha_2 e_2(\mathbf{x}) + \dots + \alpha_N e_N(\mathbf{x}) + \varepsilon(\mathbf{x}) $$ where the coefficients $\alpha_i$ are independent Gaussian r.vs with variance $\text{Var}(\alpha_i) = \lambda_i$. The noise term $\varepsilon$ has a known covariance kernel but it can be viewed as a white noise with variance $\gamma := \sum_{j > N}\lambda_j$.

When $n$ observations are made at $n$ distinct indices $\mathbf{x}_i$ the approximation takes the linear model form $$ y_i = \alpha_1 e_1(\mathbf{x}_i) + \alpha_2 e_2(\mathbf{x}_i) + \dots + \alpha_N e_N(\mathbf{x}_i) + \varepsilon_i \qquad 1 \leqslant i \leqslant n $$ or in matrix form $\mathbf{y} = \mathbf{E} \,\boldsymbol{\alpha} + \boldsymbol{\varepsilon}$ where $\mathbf{E}$ is the $n \times N$ design matrix having the eigenfunctions as its columns. We have an informative prior on $\boldsymbol{\alpha}$ namely $\boldsymbol{\alpha} \sim \text{Norm}(\mathbf{0}, \, \boldsymbol{\Delta})$ with $\boldsymbol{\Delta} := \text{diag}\{\lambda_j\}$. We can derive predictions using the Bayesian Linear Model.

Note that the related approximation for the $n \times n$ Gram matrix $\mathbf{K}$ is
$$ \mathbf{K} \approx \mathbf{E}\boldsymbol{\Delta}\mathbf{E}^\top + \gamma \mathbf{I} $$ and the relation between the Kriging prediction and the Bayesian Linear Model one can be made clear by using the matrix inversion lemma. Of course if $\gamma$ is taken as zero this will no longer work. Interestingly we can take $N > n$ if wanted. Note that the approximation will not lead to an interpolation as in exact Kriging but will be very close to it, with less numerical problems.

  • $\begingroup$ Thank you for your answer. So, If I understand correctly, as we $N$ increases, $\gamma \to 0$, right ? $\endgroup$
    – Akusa
    Apr 28, 2021 at 10:30
  • 1
    $\begingroup$ Yes because the series $\sum_j \lambda_j$ has a finite sum, nemaly the variance. $\endgroup$
    – Yves
    Apr 28, 2021 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.