Mercer's theorem and eigenfunctions Consider a kernel $K$ which satisfies the conditions of Mercer's theorem. We know that $K(x,y) = \sum\limits_{i=1}^{+\infty}\lambda_ie_i(x)e_i(y)$ where $\lambda_i$ and $e_i$ are the eigenvalues/eigenfuncions associated with the integral operator $T_K$ of $K$. Now, consider for example $K(x,y) = e^{-c|x-y|}$, that is the exponential kernel. From Stochastic Finite Elements: A Spectral Approach written by Ghanem and Spanos (Stochastic Finite Elements: A Spectral Approach), we know an expression of the eigenvalues/eigenfunctions (see p.28/29). Consider an approximation of the kernel by $K_{N}(x,y) = \sum\limits_{i=1}^{N}\lambda_ie_i(x)e_i(y)$ and we know from Mercer's theorem that $K_{N}$ converges uniformly to $K(x,y)$ as $N\to+\infty$. Take the first eigenvalue/eigenfunction of $T_K$, that is $K_{1}(x,y) = \lambda_1e_1(x)e_1(y)$. From Ghanem's book, the expression of is $e_1(x) = \frac{cos(\omega x)}{\sqrt{a + \frac{sin(2\omega a)}{2\omega}}}$ where $\omega$ is the first solution of $c - \omega tan(\omega a)= 0 $.
Finally, take 3 points $x_0 = -1, x_1 = 0 ,x_2 = 1$ and form the Gram matrix $K_{ij} = \lambda_1e_1(x_i)e_1(x_j)$ and you want to us this Gram matrix for regression. One knows that the value of at an unobserved point $x_*$ is $f(x_*) = K_1(x_*, X)K_1(X,X)^{-1}Y$ where $X = (x_0, x_1, x_2)$. The issue is that $K_1(X,X)$ is not invertible as the first row is equal to the third row and so the determinant of $K_1(X,X)$ is equal to 0 but $K(X,X)$ is invertible. What am I missing ?
 A: The problem is that in your derivation you omit the ``truncation noise'' which
must be taken into account.
Mercer's theorem can be used to get a finite-rank approximation of a
kernel. In terms of model, the Gaussian Process $Y(\mathbf{x})$ is approximated by
$$
   Y(\mathbf{x})  = \alpha_1 e_1(\mathbf{x}) + \alpha_2 e_2(\mathbf{x}) + \dots 
   + \alpha_N e_N(\mathbf{x}) + \varepsilon(\mathbf{x})
$$
where the coefficients $\alpha_i$ are independent Gaussian r.vs with variance
$\text{Var}(\alpha_i) = \lambda_i$. The noise term $\varepsilon$ has a known
covariance kernel but it can be viewed as a white noise with variance
$\gamma := \sum_{j > N}\lambda_j$.
When $n$ observations are made at $n$ distinct indices $\mathbf{x}_i$ the
approximation takes the linear model form
$$
   y_i  = \alpha_1 e_1(\mathbf{x}_i) + \alpha_2 e_2(\mathbf{x}_i) + \dots 
   + \alpha_N e_N(\mathbf{x}_i) + \varepsilon_i \qquad 1 \leqslant i
   \leqslant n
$$
or in matrix form
$\mathbf{y} = \mathbf{E} \,\boldsymbol{\alpha} +
\boldsymbol{\varepsilon}$
where $\mathbf{E}$ is the $n \times N$ design matrix having the
eigenfunctions as its columns.  We have an informative prior on
$\boldsymbol{\alpha}$ namely
$\boldsymbol{\alpha} \sim \text{Norm}(\mathbf{0}, \,
\boldsymbol{\Delta})$
with $\boldsymbol{\Delta} := \text{diag}\{\lambda_j\}$. We can derive
predictions using the Bayesian Linear Model.
Note that the related approximation for the $n \times n$ Gram matrix
$\mathbf{K}$ is
$$
   \mathbf{K} \approx  \mathbf{E}\boldsymbol{\Delta}\mathbf{E}^\top + \gamma \mathbf{I}
$$
and the relation between the Kriging prediction and the Bayesian
Linear Model one can be made clear by using the matrix inversion
lemma. Of
course if $\gamma$ is taken as zero this will no longer
work. Interestingly we can take $N > n$ if wanted. Note that the
approximation will not lead to an interpolation as in exact Kriging but will
be very close to it, with less numerical problems.
