I want to know the relationship between the eigensystems of two non-negative-definite (covariance-like) matrices. Both are derived from X which is a T-by-K real matrix (wlog say K > T). I avoid subtracting the mean to avoid complicating the main issue which is about the eigensystem structure.

Ct = (1/T) * X' X = Et Lt Et'
Ck = (1/K) * X X' = Ek Lk Ek'

.... where Et, Lt are the (column) eigenvectors and diagonal matrix of eigenvalues respectively for Ct, and Ek Lk are the same for Ck.

I remember vaguely from school that there is some kind of duality relationship here (between the two eigensystems) which is particularly interesting because both matrices have rank T < K, and hence only T of the diagonal elements of Lt are nonzero.

Does anyone know this relationship explicitly? Thanks in advance for any light you can shed!

  • 2
    $\begingroup$ Please use mathjax formatting for formulae and expressions in your question. A guide on how to do so can be found here. meta.math.stackexchange.com/questions/5020/… $\endgroup$
    – Sycorax
    May 27 '15 at 18:50
  • $\begingroup$ Covariance matrices are not computed this way. The variables are first centered to have zero mean. This complicates the relationships between the eigenvalues of the two matrices. So, what are you actually asking about: covariance matrices or the simpler sum-of-products matrices given in the formulas? $\endgroup$
    – whuber
    May 27 '15 at 19:20
  • $\begingroup$ Good point, assume we leave the mean in; this is more a question about the relation of the eigensystems. $\endgroup$
    – Axe Duck
    May 27 '15 at 20:10
  • $\begingroup$ Note: you shouldn't say "non-negative-definite" when you mean "positive semi-definite", those are very different things (the matrix may simply have no definiteness at all, for example, i.e., have both positive and negative eigenvalues). $\endgroup$
    – cfh
    May 27 '15 at 22:41

This is most readily analyzed by means of the singular value decomposition

$$X = U \Sigma V^\prime$$

where the columns of $U$ are mutually orthonormal, so are the columns of $V$, and $\Sigma$ has nonzero values only on its diagonal $\Sigma_{ii}$. This immediately yields the decompositions

$$X^\prime X = V\Sigma^\prime\Sigma V^\prime$$


$$X X^\prime = U \Sigma\Sigma^\prime U^\prime.$$

Since $V^{-1} = V^\prime$ and $U^{-1}=U^\prime$ and both $\Sigma^\prime\Sigma$ and $\Sigma\Sigma^\prime$ are square diagonal matrices, both of these equations are eigendecompositions. (That demonstrates $\Sigma$ is uniquely determined by $X$ up to a discrete set of permutations.) The diagonal elements of these latter two matrices--which are the eigenvalues--are the set of $\Sigma_{ii}^2$, with the same multiplicities. When $X$ is not square, the larger of $\Sigma^\prime\Sigma$ and $\Sigma\Sigma^\prime$ will have additional zeros to complete its diagonal.

There can be no universal relationship between $U$ (whose columns are eigenvectors of $XX^\prime$) and $V$ (whose columns are eigenvectors of $X^\prime X$), because we could start with arbitrary matrices $U$, $\Sigma$, and $V$ satisfying the SVD requirements and construct an example $X = U\Sigma V^\prime$ having these particular, but arbitrary, $U$ and $V$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.