Mercer's theorem and eigenfunctions

Question

Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/eigenfuncions associated with the integral operator $T_K$ of $K$. Now, consider for example $K(x,y) = e^{-c|x-y|}$, that is the exponential kernel. From Stochastic Finite Elements: A Spectral Approach written by Ghanem and Spanos (Stochastic Finite Elements: A Spectral Approach), we know an expression of the eigenvalues/eigenfunctions (see p.28/29). Consider an approximation of the kernel by $K_{N}(x,y) = \sum\limits_{i=1}^{N}\lambda_ie_i(x)e_i(y)$ and we know from Mercer's theorem that $K_{N}$ converges uniformly to $K(x,y)$ as $N\to+\infty$. Take the first eigenvalue/eigenfunction of $T_K$, that is $K_{1}(x,y) = \lambda_1e_1(x)e_1(y)$. From Ghanem's book, the expression of is $e_1(x) = \frac{cos(\omega x)}{\sqrt{a + \frac{sin(2\omega a)}{2\omega}}}$ where $\omega$ is the first solution of $c - \omega tan(\omega a)= 0 $.

Finally, take 3 points $x_0 = -1, x_1 = 0 ,x_2 = 1$ and form the Gram matrix $K_{ij} = \lambda_1e_1(x_i)e_1(x_j)$ and you want to us this Gram matrix for regression. One knows that the value of at an unobserved point $x_*$ is $f(x_*) = K_1(x_*, X)K_1(X,X)^{-1}Y$ where $X = (x_0, x_1, x_2)$. The issue is that $K_1(X,X)$ is not invertible as the first row is equal to the third row and so the determinant of $K_1(X,X)$ is equal to 0 but $K(X,X)$ is invertible. What am I missing ?

Answer 1

The problem is that in your derivation you omit the ``truncation noise'' which must be taken into account.

Mercer's theorem can be used to get a finite-rank approximation of a kernel. In terms of model, the Gaussian Process $Y(\mathbf{x})$ is approximated by $$ Y(\mathbf{x}) = \alpha_1 e_1(\mathbf{x}) + \alpha_2 e_2(\mathbf{x}) + \dots + \alpha_N e_N(\mathbf{x}) + \varepsilon(\mathbf{x}) $$ where the coefficients $\alpha_i$ are independent Gaussian r.vs with variance $\text{Var}(\alpha_i) = \lambda_i$. The noise term $\varepsilon$ has a known covariance kernel but it can be viewed as a white noise with variance $\gamma := \sum_{j > N}\lambda_j$.

When $n$ observations are made at $n$ distinct indices $\mathbf{x}_i$ the approximation takes the linear model form $$ y_i = \alpha_1 e_1(\mathbf{x}_i) + \alpha_2 e_2(\mathbf{x}_i) + \dots + \alpha_N e_N(\mathbf{x}_i) + \varepsilon_i \qquad 1 \leqslant i \leqslant n $$ or in matrix form $\mathbf{y} = \mathbf{E} \,\boldsymbol{\alpha} + \boldsymbol{\varepsilon}$ where $\mathbf{E}$ is the $n \times N$ design matrix having the eigenfunctions as its columns. We have an informative prior on $\boldsymbol{\alpha}$ namely $\boldsymbol{\alpha} \sim \text{Norm}(\mathbf{0}, \, \boldsymbol{\Delta})$ with $\boldsymbol{\Delta} := \text{diag}\{\lambda_j\}$. We can derive predictions using the Bayesian Linear Model.

Note that the related approximation for the $n \times n$ Gram matrix $\mathbf{K}$ is
$$ \mathbf{K} \approx \mathbf{E}\boldsymbol{\Delta}\mathbf{E}^\top + \gamma \mathbf{I} $$ and the relation between the Kriging prediction and the Bayesian Linear Model one can be made clear by using the matrix inversion lemma. Of course if $\gamma$ is taken as zero this will no longer work. Interestingly we can take $N > n$ if wanted. Note that the approximation will not lead to an interpolation as in exact Kriging but will be very close to it, with less numerical problems.