# Mercer's theorem and eigenfunctions

Consider a kernel $$K$$ which satisfies the conditions of Mercer's theorem. We know that $$K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$$ where $$\lambda_i$$ and $$e_i$$ are the eigenvalues/eigenfuncions associated with the integral operator $$T_K$$ of $$K$$. Now, consider for example $$K(x,y) = e^{-c|x-y|}$$, that is the exponential kernel. From Stochastic Finite Elements: A Spectral Approach written by Ghanem and Spanos (Stochastic Finite Elements: A Spectral Approach), we know an expression of the eigenvalues/eigenfunctions (see p.28/29). Consider an approximation of the kernel by $$K_{N}(x,y) = \sum\limits_{i=1}^{N}\lambda_ie_i(x)e_i(y)$$ and we know from Mercer's theorem that $$K_{N}$$ converges uniformly to $$K(x,y)$$ as $$N\to+\infty$$. Take the first eigenvalue/eigenfunction of $$T_K$$, that is $$K_{1}(x,y) = \lambda_1e_1(x)e_1(y)$$. From Ghanem's book, the expression of is $$e_1(x) = \frac{cos(\omega x)}{\sqrt{a + \frac{sin(2\omega a)}{2\omega}}}$$ where $$\omega$$ is the first solution of $$c - \omega tan(\omega a)= 0$$.

Finally, take 3 points $$x_0 = -1, x_1 = 0 ,x_2 = 1$$ and form the Gram matrix $$K_{ij} = \lambda_1e_1(x_i)e_1(x_j)$$ and you want to us this Gram matrix for regression. One knows that the value of at an unobserved point $$x_*$$ is $$f(x_*) = K_1(x_*, X)K_1(X,X)^{-1}Y$$ where $$X = (x_0, x_1, x_2)$$. The issue is that $$K_1(X,X)$$ is not invertible as the first row is equal to the third row and so the determinant of $$K_1(X,X)$$ is equal to 0 but $$K(X,X)$$ is invertible. What am I missing ?

The problem is that in your derivation you omit the truncation noise'' which must be taken into account.
Mercer's theorem can be used to get a finite-rank approximation of a kernel. In terms of model, the Gaussian Process $$Y(\mathbf{x})$$ is approximated by $$Y(\mathbf{x}) = \alpha_1 e_1(\mathbf{x}) + \alpha_2 e_2(\mathbf{x}) + \dots + \alpha_N e_N(\mathbf{x}) + \varepsilon(\mathbf{x})$$ where the coefficients $$\alpha_i$$ are independent Gaussian r.vs with variance $$\text{Var}(\alpha_i) = \lambda_i$$. The noise term $$\varepsilon$$ has a known covariance kernel but it can be viewed as a white noise with variance $$\gamma := \sum_{j > N}\lambda_j$$.
When $$n$$ observations are made at $$n$$ distinct indices $$\mathbf{x}_i$$ the approximation takes the linear model form $$y_i = \alpha_1 e_1(\mathbf{x}_i) + \alpha_2 e_2(\mathbf{x}_i) + \dots + \alpha_N e_N(\mathbf{x}_i) + \varepsilon_i \qquad 1 \leqslant i \leqslant n$$ or in matrix form $$\mathbf{y} = \mathbf{E} \,\boldsymbol{\alpha} + \boldsymbol{\varepsilon}$$ where $$\mathbf{E}$$ is the $$n \times N$$ design matrix having the eigenfunctions as its columns. We have an informative prior on $$\boldsymbol{\alpha}$$ namely $$\boldsymbol{\alpha} \sim \text{Norm}(\mathbf{0}, \, \boldsymbol{\Delta})$$ with $$\boldsymbol{\Delta} := \text{diag}\{\lambda_j\}$$. We can derive predictions using the Bayesian Linear Model.
Note that the related approximation for the $$n \times n$$ Gram matrix $$\mathbf{K}$$ is
$$\mathbf{K} \approx \mathbf{E}\boldsymbol{\Delta}\mathbf{E}^\top + \gamma \mathbf{I}$$ and the relation between the Kriging prediction and the Bayesian Linear Model one can be made clear by using the matrix inversion lemma. Of course if $$\gamma$$ is taken as zero this will no longer work. Interestingly we can take $$N > n$$ if wanted. Note that the approximation will not lead to an interpolation as in exact Kriging but will be very close to it, with less numerical problems.
• Thank you for your answer. So, If I understand correctly, as we $N$ increases, $\gamma \to 0$, right ? – Akusa Apr 28 at 10:30
• Yes because the series $\sum_j \lambda_j$ has a finite sum, nemaly the variance. – Yves Apr 28 at 10:35