# On the properties of covariance and kernel matrices

I'm stumbling upon an example of a mixed model or a Gaussian Process, say:

• $$Z \in\mathbb{R}^{n \times m}, m \ge n$$ ie random effect
• $$X \in\mathbb{R}^{n \times p}, p \ge 1$$ ie fixed effects
• $$K \in\mathbb{R}^{n \times n}, K = ZZ^{T}$$, ie. a linear kernel and Hermitian positive semidifinte
• $$I \in\mathbb{R}^{n \times n}$$ being the identity matrix
• $$\Sigma \in\mathbb{R}^{n \times n}, \Sigma = \sigma_{1}^{2}K + \sigma_{2}^{2}I$$ where $$\Sigma$$ is a covariance matrix and Hermitian positive definite
• $$P=X(X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1}$$ is a non-symmetric projection matrix
• $$R = \Sigma^{-1}(I - P)$$

Putting aside what we know about covariances etc, we can say $$\Sigma$$ is a kernel, since it's made up of adding two valid kernels.

I have the following questions:

1. What can we say about $$R$$, it is a symmetric positive definite matrix, and also a valid kernel itself?

2. What about $$\Sigma^{-1}$$, can we say this is a valid kernel too? This should be true, since it's just the precision matrix, and inverse of a hermitian is hermitian itself, right?

3. And finally, if 2 is true, for $$A = \Sigma^{-1}K$$ or $$A = \Sigma^{-1}K_{2}$$ where $$K_{2}$$ is another kernel, A is also a valid kernel hence symmetric positive semidefinite?

For 3, I'm assuming if 2 holds, then 3 is just the multiplication of two kernels hence holds true.

1. Correct, $$\Sigma^{-1}(I-P)$$, positive definite, would also be a kernel.
2. Correct, $$\Sigma^{-1}$$, positive definite is a valid kernel.
3. $$\Sigma^{-1} K = \frac{1}{\sigma^2_1}I-\frac{\sigma^2_2}{\sigma^2_1}\Sigma^{-1}$$ generally is not a valid kernel matrix. It is still symmetric, but not necessarily positive definite, whereas for a general kernel matrix $$K_2$$ even the symmetry of $$\Sigma^{-1}K_2$$ is not guaranteed.

Reminder:

The only requirement for a matrix to be useful as a kernel matrix is whether one can construct a valid inner product map with it.

Restricting ourselves to vector space $$\mathbb{R}^n$$, an inner product would be a map $$x,y \to x^T K y$$ with a matrix $$K \in \mathbb{R}^{n\times n}$$ such that $$\forall x,y,z \in \mathbb{R}^n$$

1. (symmetry) $$x^T K y = y^T K x$$
2. (linearity) $$(\alpha x)^T K y = \alpha x^T K y$$ and $$(x+z)^T K y = x^T K y + z^T K y$$
3. (positive-definiteness) $$x^T K x \geq 0$$ and zero only if the argument $$x$$ is zero (i.e. $$x = \mathbf{0}$$).

Wherefrom stem the requirements for positive-definiteness of kernel matrix $$K$$.

• +1 cool! Which identity did you use for 3? Would you please include the derivation?
– NULL
Commented Nov 4, 2019 at 19:40
• Couple of follow up questions before accepting the answer, 1- For $B = \Sigma^{-1}K_{2}$ the lack of guarantee comes from the fact that we don't have a way to guess the properties of $B$ without knowing if $\Sigma^{-1}$ and $K_{2}$ commute? Then can't we claim this for any multiplication of kernels? Because I see it is very common to multiply kernels and assume it's a valid kernel, ie. simple example would be polynomial degree 2 from linear etc. but of course there are many more scenarios?
– NULL
Commented Nov 4, 2019 at 19:44
• And the second question is what can we say about $V=RK$? Based on 3, $V=aI - b\Sigma^{-1} - b\Sigma^{-1}X(X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1} - a\Sigma^{-1}X(X^{T}\Sigma^{-1}X)^{-1}X^{T}$, $a = \frac{1}{\sigma^2_1}$ , $b=\frac{\sigma^2_2}{\sigma^2_1}$ hence non-symmetric?
– NULL
Commented Nov 4, 2019 at 19:50
• Hey, a) For 3 I just solved your definition of $\Sigma$ for $K$. Do you want me to develop that? Regarding your questions: b) when they speak of kernel multiplication, the multiplication of kernel functions $K(x,y)$ is meant, not the kernel matrices product. So either indeed you take two kernel matrices that commute or you may obtain a kernel matrix from two valid kernel matrices by means of Schur (aka Hadamard aka elementwise) product which again produces a positive-definite matrix. c) $RK$ is not valid for the same reason, you cannot guarantee its positive-definiteness. Commented Nov 4, 2019 at 21:44
• You can decompose $\Sigma^{-1} = C C^T$ due to its symmetry and positive-definiteness. Then $a^T R a = a^TC ( I - C^T X (X^T C C^T X)^{-1} X^T C ) C^T a \geq 0$, because expression in the middle is again a projection (and therefore psd) matrix. Commented Nov 6, 2019 at 22:29