7
$\begingroup$

I understand that Multidimensional scaling (MDS) is same as doing Principal Components analysis (PCA) if Euclidean distance is used, this is known as Metric MDS. But I came across this in a book that "it has been shown (Chatfield and Collins 1980) that the eigenvalues of $XX^T$ (unnormalised outer product matrix) are equal to the eigenvalues $X^TX$ (unnormalised inner product matrix) and eigenvectors of $XX^T$ and $X^TX$ are related by a linear transformation. "

Note, that X denotes a matrix of data, with $n$ features (rows) and $m$ instances (columns).

Now I am unable to get this Chattield and Collins book anywhere, and I can understand that the eigenvalues are equal. But how are the eigenvectors of PCA and metric MDS related to each other ?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ When you left-multiply the eigenvector equation $(XX')b=\lambda b$ by $X'$, factor the scalar $\lambda$ through, and write $e=X'b$, you get $(X'X)e=\lambda e$, immediately obtaining the Chatfield & Collins results. $\endgroup$
    – whuber ♦
    Jun 3, 2013 at 14:25
  • $\begingroup$ Left multiplying by $\mathbf{X}^T$, I obtain $$\mathbf{X}^T \mathbf{X} \mathbf{X}^T \mathbf{b} = \lambda \mathbf{b}$$. After this, what happens ? What do you mean by "factor the scalar $\lambda$ through ? $\endgroup$
    – user76170
    Nov 4, 2013 at 10:29
  • $\begingroup$ When the ground ring is commutative--as it is when you are using real or complex numbers--the order of multiplication does not matter. That fact will be needed to simplify the right hand side after you remember that it, too, needs to be left-multiplied by $X'$ in order to maintain the equality. $\endgroup$
    – whuber ♦
    Nov 4, 2013 at 20:16

1 Answer 1

Reset to default
7
$\begingroup$

I find it helpful to consider the singular value decomposition for questions like this with the assumption that $X$ is a real matrix. Writing $X = UDV^T$, we can see that $XX^T = UD^2U^T$ and $X^TX = VD^2V^T$. As we can see, the eigenvalues of both $XX^T$ and $X^TX$ are contained in the diagonal matrix $D^2$ and are indeed equal. Also, we see that the matrix of eigenvectors of $XX^T$ is $U$, while the matrix of eigenvectors of $X^TX$ is $V$.

Because $$ X^TU = VDU^TU = VD, $$ we have the relationship $X^TU = VD$, similar to that pointed out by whuber.

Assuming $D$ is nonsingular, two additional properties that prove to be quite useful are:

$$ \begin{align} U &= XVD^{-1}\\ V &= X^TUD^{-1}. \end{align} $$

Improve this answer
$\endgroup$
4
  • $\begingroup$ @ John : I understood that till the matrix of eigenvectors part, but after that how do you conclude $X^TU = VD$ ? $\endgroup$
    – user76170
    Jun 19, 2013 at 16:37
  • $\begingroup$ I updated my answer to clarify the conclusion. $\endgroup$
    – ramhiser
    Jun 19, 2013 at 21:21
  • $\begingroup$ I understood your derivation of $\mathbf{X}^T \mathbf{U} = \mathbf{V} \mathbf{D}$, so we can write $\mathbf{U} = {(\mathbf{X}^T})^{-1} \mathbf{V} \mathbf{D}$. But now I am again stuck, this eqn is fine theoretically, but practically how do you find ${\mathbf{X}^T}^{-1}$, since $\mathbf{X}$ is rectangular. Surely, there must be some other way. $\endgroup$
    – user76170
    Nov 4, 2013 at 10:43
  • $\begingroup$ Because $X$ is rectangular, it does not have an inverse. I have updated my answer to answer your follow-up question. To calculate $U$, we start with $XV = UD$ and postmultiply each side by $D^{-1}$. $\endgroup$
    – ramhiser
    Nov 4, 2013 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.