7
$\begingroup$

For a $n\times p$ matrix $X$, where $p \gg n$, what is the relationship between $X^{T}X$ (scatter matrix, on which covariance matrix is based) and $XX^{T}$ (outer product sometimes called Gram matrix)?

If one is known, how is it possible to obtain the other (the best one can do)?

$\endgroup$
  • $\begingroup$ Please review my edit of your question, will it be all right for you? $\endgroup$ – ttnphns Aug 6 '15 at 16:37
  • $\begingroup$ @ttnphns Thanks for the edit. It is totally fine with me. $\endgroup$ – Amir Aug 6 '15 at 16:41
  • $\begingroup$ I think you swapped scatter matrix and Gram matrix. Scatter (or cov) matrix should be XX' and Gram matrix should be X'X. $\endgroup$ – PhABC Apr 17 '17 at 16:47
9
$\begingroup$

A Singular Value Decomposition (SVD) of $X$ expresses it as

$$X = U D V^\prime$$

where $U$ is an $n\times r$ matrix whose columns are mutually orthonormal, $V$ is an $p\times r$ matrix whose columns are mutually orthonormal, and $D$ is an $r\times r$ diagonal matrix with positive values (the "singular values" of $X$) on the diagonal. Necessarily $r$--which is the rank of $X$--can be no greater than either $n$ or $p$.

Using this we compute

$$X^\prime X = (U D V^\prime)^\prime U D V^\prime = V D^\prime U^\prime U D V^\prime = V D^2 V^\prime$$

and

$$ X X^\prime= U D V^\prime (U D V^\prime)^\prime= U D V^\prime V D^\prime U^\prime= U D^2 U^\prime.$$

Although we can recover $D^2$ by diagonalizing either of $X^\prime X$ or $X X^\prime$, the former gives no information about $U$ and the latter gives no information about $V$. However, $U$ and $V$ are completely independent of each other--starting with one of them, along with $D$, you can choose the other arbitrarily (subject to the orthonormality conditions) and construct a valid matrix $X$. Therefore $D^2$ contains all the information that is common to the matrices $X^\prime X$ and $X X^\prime$.


There is a nice geometric interpretation that helps make this convincing. The SVD allows us to view any linear transformation $T_X$ (as represented by the matrix $X$) from $\mathbb{R}^p$ to $\mathbb{R}^n$ in terms of three easily understood linear transformations:

$V$ is the matrix of a transformation $T_V:\mathbb{R}^r \to \mathbb{R}^p$ that is one-to-one (has no kernel) and isometric. That is, it rotates $\mathbb{R}^r$ into an $r$-dimensional subspace $T_V(\mathbb{R}^r)$ of a $p$-dimensional space.

$U$ similarly is the matrix of a one-to-one, isometric transformation $T_U:\mathbb{R}^r\to \mathbb{R}^n$.

$D$ positively rescales the $r$ coordinate axes in $\mathbb{R}^r$, corresponding to a linear transformation $T_D$ that distorts the unit sphere (used for reference) into an ellipsoid without rotating it.

The transpose of $V$, $V^\prime$, corresponds to a linear transformation $T_{V^\prime}:\mathbb{R}^p\to\mathbb{R}^r$ that kills all vectors in $\mathbb{R}^p$ that are perpendicular to $T_V(\mathbb{R}^r)$. It otherwise rotates $T_V(\mathbb{R}^r)$ into $\mathbb{R}^r$. Equivalently, you can think of $T_{V^\prime}$ as "ignoring" any perpendicular directions and establishing an orthonormal coordinate system within $T_V(\mathbb{R}^r) \subset \mathbb{R}^p$. $T_D$ acts directly on that coordinate system, expanding by various amounts (as specified by the singular values) along the coordinate axes determined by $V$. $T_U$ then maps the result into $\mathbb{R}^n$.

The linear transformation associated with $X^\prime X$ in effect acts on $T_V(\mathbb{R}^r)$ through two "round trips": $T_X$ expands the coordinates in the system determined by $V$ by $T_D$ and then $T_{X^\prime}$ does it all over again. Similarly, $X X^\prime$ does exactly the same thing to the $r$-dimensional subspace of $\mathbb{R}^n$ established by the $r$ orthogonal columns of $U$. Thus, the role of $V$ is to describe a frame in a subspace of $\mathbb{R}^p$ and the role of $U$ is to describe a frame in a subspace of $\mathbb{R}^n$. The matrix $X^\prime X$ gives us information about the frame in the first space and $X X\prime$ tells us the frame in the second space, but those two frames don't have to have any relationship at all to one another.

$\endgroup$
  • $\begingroup$ Very well explained. So the necessary and sufficient condition for deriving a valid (dual?) product matrix from the other is that trace of the two must be equal? $\endgroup$ – Amir Aug 6 '15 at 17:20
  • 1
    $\begingroup$ I suppose it all depends on what you mean by "deriving." I would have understood it to be solving the simultaneous equations $A=X^\prime X,B=XX^\prime$ for $X$ in terms of the positive-semidefinite symmetric matrices $A$ and $B$. If that's your meaning, then you need the diagonal parts of the SVDs of $A$ and $B$ (that is, their nonzero spectra) to be the same up to permutation: that's much stronger than the traces being equal. $\endgroup$ – whuber Aug 6 '15 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.