What is the duality relationship between eigensystems of X X' vs X' X?

Question

I want to know the relationship between the eigensystems of two non-negative-definite (covariance-like) matrices. Both are derived from X which is a T-by-K real matrix (wlog say K > T). I avoid subtracting the mean to avoid complicating the main issue which is about the eigensystem structure.

Ct = (1/T) * X' X = Et Lt Et'
Ck = (1/K) * X X' = Ek Lk Ek'

.... where Et, Lt are the (column) eigenvectors and diagonal matrix of eigenvalues respectively for Ct, and Ek Lk are the same for Ck.

I remember vaguely from school that there is some kind of duality relationship here (between the two eigensystems) which is particularly interesting because both matrices have rank T < K, and hence only T of the diagonal elements of Lt are nonzero.

Does anyone know this relationship explicitly? Thanks in advance for any light you can shed!

Answer 1

This is most readily analyzed by means of the singular value decomposition

$$X = U \Sigma V^\prime$$

where the columns of $U$ are mutually orthonormal, so are the columns of $V$, and $\Sigma$ has nonzero values only on its diagonal $\Sigma_{ii}$. This immediately yields the decompositions

$$X^\prime X = V\Sigma^\prime\Sigma V^\prime$$

and

$$X X^\prime = U \Sigma\Sigma^\prime U^\prime.$$

Since $V^{-1} = V^\prime$ and $U^{-1}=U^\prime$ and both $\Sigma^\prime\Sigma$ and $\Sigma\Sigma^\prime$ are square diagonal matrices, both of these equations are eigendecompositions. (That demonstrates $\Sigma$ is uniquely determined by $X$ up to a discrete set of permutations.) The diagonal elements of these latter two matrices--which are the eigenvalues--are the set of $\Sigma_{ii}^2$, with the same multiplicities. When $X$ is not square, the larger of $\Sigma^\prime\Sigma$ and $\Sigma\Sigma^\prime$ will have additional zeros to complete its diagonal.

There can be no universal relationship between $U$ (whose columns are eigenvectors of $XX^\prime$) and $V$ (whose columns are eigenvectors of $X^\prime X$), because we could start with arbitrary matrices $U$, $\Sigma$, and $V$ satisfying the SVD requirements and construct an example $X = U\Sigma V^\prime$ having these particular, but arbitrary, $U$ and $V$.