On the properties of covariance and kernel matrices I'm stumbling upon an example of a mixed model or a Gaussian Process, say:


*

*$Z \in\mathbb{R}^{n \times m}, m \ge n$ ie random effect

*$X \in\mathbb{R}^{n \times p}, p \ge 1$ ie fixed effects

*$K \in\mathbb{R}^{n \times n}, K = ZZ^{T}$, ie. a linear kernel and Hermitian positive semidifinte

*$I \in\mathbb{R}^{n \times n}$ being the identity matrix

*$\Sigma \in\mathbb{R}^{n \times n}, \Sigma = \sigma_{1}^{2}K +  \sigma_{2}^{2}I$ where $\Sigma$ is a covariance matrix and Hermitian positive definite

*$P=X(X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1}$ is a non-symmetric projection matrix

*$R = \Sigma^{-1}(I - P)$
Putting aside what we know about covariances etc, we can say $\Sigma$ is a kernel, since it's made up of adding two valid kernels.
I have the following questions:


*

*What can we say about $R$, it is a symmetric positive definite matrix, and also a valid kernel itself?

*What about $\Sigma^{-1}$, can we say this is a valid kernel too? This should be true, since it's just the precision matrix, and inverse of a hermitian is hermitian itself, right?

*And finally, if 2 is true, for $A = \Sigma^{-1}K$ or $A = \Sigma^{-1}K_{2}$ where $K_{2}$ is another kernel, A is also a valid kernel hence symmetric positive semidefinite?
For 3, I'm assuming if 2 holds, then 3 is just the multiplication of two kernels hence holds true.
 A: Answer:


*

*Correct, $\Sigma^{-1}(I-P)$, positive definite, would also be a kernel. 

*Correct, $\Sigma^{-1}$, positive definite is a valid kernel.

*$\Sigma^{-1} K = \frac{1}{\sigma^2_1}I-\frac{\sigma^2_2}{\sigma^2_1}\Sigma^{-1}$ generally is not a valid kernel matrix. It is still symmetric, but not necessarily positive definite, whereas for a general kernel matrix $K_2$ even the symmetry of $\Sigma^{-1}K_2$ is not guaranteed.


Reminder:
The only requirement for a matrix to be useful as a kernel matrix is whether one can construct a valid inner product map with it.
Restricting ourselves to vector space $\mathbb{R}^n$, an inner product would be a map $x,y \to x^T K y$ with a matrix $K \in \mathbb{R}^{n\times n}$ such that $\forall x,y,z \in \mathbb{R}^n$


*

*(symmetry) $ x^T K y = y^T K x$

*(linearity) $ (\alpha x)^T K y = \alpha x^T K y $ and $ (x+z)^T K y = x^T K y + z^T K y$

*(positive-definiteness) $x^T K x \geq 0 $ and zero only if the argument $x$ is zero (i.e. $x = \mathbf{0}$).


Wherefrom stem the requirements for positive-definiteness of kernel matrix $K$.
