2
$\begingroup$

I'm stumbling upon an example of a mixed model or a Gaussian Process, say:

  • $Z \in\mathbb{R}^{n \times m}, m \ge n$ ie random effect
  • $X \in\mathbb{R}^{n \times p}, p \ge 1$ ie fixed effects
  • $K \in\mathbb{R}^{n \times n}, K = ZZ^{T}$, ie. a linear kernel and Hermitian positive semidifinte
  • $I \in\mathbb{R}^{n \times n}$ being the identity matrix
  • $\Sigma \in\mathbb{R}^{n \times n}, \Sigma = \sigma_{1}^{2}K + \sigma_{2}^{2}I$ where $\Sigma$ is a covariance matrix and Hermitian positive definite
  • $P=X(X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1}$ is a non-symmetric projection matrix
  • $R = \Sigma^{-1}(I - P)$

Putting aside what we know about covariances etc, we can say $\Sigma$ is a kernel, since it's made up of adding two valid kernels.

I have the following questions:

  1. What can we say about $R$, it is a symmetric positive definite matrix, and also a valid kernel itself?

  2. What about $\Sigma^{-1}$, can we say this is a valid kernel too? This should be true, since it's just the precision matrix, and inverse of a hermitian is hermitian itself, right?

  3. And finally, if 2 is true, for $A = \Sigma^{-1}K$ or $A = \Sigma^{-1}K_{2}$ where $K_{2}$ is another kernel, A is also a valid kernel hence symmetric positive semidefinite?

For 3, I'm assuming if 2 holds, then 3 is just the multiplication of two kernels hence holds true.

$\endgroup$
0

1 Answer 1

5
+100
$\begingroup$

Answer:

  1. Correct, $\Sigma^{-1}(I-P)$, positive definite, would also be a kernel.
  2. Correct, $\Sigma^{-1}$, positive definite is a valid kernel.
  3. $\Sigma^{-1} K = \frac{1}{\sigma^2_1}I-\frac{\sigma^2_2}{\sigma^2_1}\Sigma^{-1}$ generally is not a valid kernel matrix. It is still symmetric, but not necessarily positive definite, whereas for a general kernel matrix $K_2$ even the symmetry of $\Sigma^{-1}K_2$ is not guaranteed.

Reminder:

The only requirement for a matrix to be useful as a kernel matrix is whether one can construct a valid inner product map with it.

Restricting ourselves to vector space $\mathbb{R}^n$, an inner product would be a map $x,y \to x^T K y$ with a matrix $K \in \mathbb{R}^{n\times n}$ such that $\forall x,y,z \in \mathbb{R}^n$

  1. (symmetry) $ x^T K y = y^T K x$
  2. (linearity) $ (\alpha x)^T K y = \alpha x^T K y $ and $ (x+z)^T K y = x^T K y + z^T K y$
  3. (positive-definiteness) $x^T K x \geq 0 $ and zero only if the argument $x$ is zero (i.e. $x = \mathbf{0}$).

Wherefrom stem the requirements for positive-definiteness of kernel matrix $K$.

$\endgroup$
7
  • $\begingroup$ +1 cool! Which identity did you use for 3? Would you please include the derivation? $\endgroup$
    – NULL
    Commented Nov 4, 2019 at 19:40
  • $\begingroup$ Couple of follow up questions before accepting the answer, 1- For $B = \Sigma^{-1}K_{2}$ the lack of guarantee comes from the fact that we don't have a way to guess the properties of $B$ without knowing if $\Sigma^{-1}$ and $K_{2}$ commute? Then can't we claim this for any multiplication of kernels? Because I see it is very common to multiply kernels and assume it's a valid kernel, ie. simple example would be polynomial degree 2 from linear etc. but of course there are many more scenarios? $\endgroup$
    – NULL
    Commented Nov 4, 2019 at 19:44
  • $\begingroup$ And the second question is what can we say about $V=RK$? Based on 3, $V=aI - b\Sigma^{-1} - b\Sigma^{-1}X(X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1} - a\Sigma^{-1}X(X^{T}\Sigma^{-1}X)^{-1}X^{T}$, $a = \frac{1}{\sigma^2_1}$ , $b=\frac{\sigma^2_2}{\sigma^2_1}$ hence non-symmetric? $\endgroup$
    – NULL
    Commented Nov 4, 2019 at 19:50
  • 1
    $\begingroup$ Hey, a) For 3 I just solved your definition of $\Sigma$ for $K$. Do you want me to develop that? Regarding your questions: b) when they speak of kernel multiplication, the multiplication of kernel functions $K(x,y)$ is meant, not the kernel matrices product. So either indeed you take two kernel matrices that commute or you may obtain a kernel matrix from two valid kernel matrices by means of Schur (aka Hadamard aka elementwise) product which again produces a positive-definite matrix. c) $RK$ is not valid for the same reason, you cannot guarantee its positive-definiteness. $\endgroup$
    – Konstantin
    Commented Nov 4, 2019 at 21:44
  • 1
    $\begingroup$ You can decompose $\Sigma^{-1} = C C^T$ due to its symmetry and positive-definiteness. Then $a^T R a = a^TC ( I - C^T X (X^T C C^T X)^{-1} X^T C ) C^T a \geq 0$, because expression in the middle is again a projection (and therefore psd) matrix. $\endgroup$
    – Konstantin
    Commented Nov 6, 2019 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.