What's the difference between principal components analysis and multidimensional scaling?

Question

How are PCA and classical MDS different? How about MDS versus nonmetric-MDS? Is there a time when you would prefer one over the other? How do the interpretations differ?

Answer 1

Classic Torgerson metric MDS is actually done by transforming distances into similarities and performing PCA (eigen-decomposition or singular-value-decomposition) on those. [The other name of this procedure is Principal Coordinate Analysis.] So, PCA might be called the algorithm of the simplest MDS.

Non-metric MDS is based on iterative ALSCAL algorithm (or algorithm similar to it) which is a more versatile mapping technique than PCA and can be applied to metric MDS as well. While PCA retains m important dimensions for you, ALSCAL fits configuration to m dimensions (you pre-define m).

Thus, MDS and PCA are not at the same level to be in line or opposite to each other. PCA is just a method while MDS is a class of analysis. As mapping, PCA is a particular case of MDS. On the other hand, PCA is a particular case of Factor analysis which, being a data reduction, is more than only a mapping, while MDS is only a mapping.

As for your question about metric MDS vs non-metric MDS there's little to comment because the answer is straightforward. If I believe my input dissimilarities are so close to be euclidean distances that a linear transform will suffice to map them in m-dimensional space, I will prefer metric MDS. If I don't believe, then monotonic transform is necessary, implying use of non-metric MDS.

Answer 2

Uhm... quite different. In PCA, you are given the multivariate continuous data (a multivariate vector for each subject), and you are trying to figure out if you don't need that many dimensions to conceptualize them. In (metric) MDS, you are given the matrix of distances between the objects, and you are trying to figure out what the locations of these objects in space are (and whether you need a 1D, 2D, 3D, etc. space). In non-metric MDS, you only know that objects 1 and 2 are more distant than objects 2 and 3, so you try to quantify that, on top of finding the dimensions and locations.

With a notable stretch of imagination, you can say that a common goal of PCA and MDS is to visualize objects in 2D or 3D. But given how different the inputs are, these methods won't be discussed as even distantly related in any multivariate textbook. I would guess that you can convert the data usable for PCA into data usable for MDS (say, by computing Mahalanobis distances between objects, using the sample covariance matrix), but that would immediately result in a loss of information: MDS is only defined up to location and rotation, and the latter two can be done more informatively with PCA.

Answer 3

PCA yields the EXACT same results as classical MDS -IF- Euclidean distance is used.

I'm quoting Cox & Cox (2001), p 43-44 "There is a duality between a principals components analysis and PCO [principal coordinates analysis, aka classical MDS] where dissimilarities are given by Euclidean distance."

The section in Cox & Cox explains it pretty clearly:

Imagine you have $X$ = attributes of $n$ products by $p$ dimensions, mean centered

PCA is attained by finding eigenvectors of the covariance matrix ~ $X'X$ (divided by n-1) -- call the eigenvectors $\xi$, and eigenvalues $\mu$.

MDS is attained by first converting $X$ into distance matrix, here, Euclidean distance, i.e., $XX'$, then finding the eigenvectors -- call the eigenvectors $v$, and eigenvalues $\lambda$.

p 43: "It is a well known result that the eigenvalues of $XX'$ are the same as those for $X'X$, together with an extra n-p zero eigenvalues." So, for $i < p$, $\mu_i$ = $\lambda_i$

Going back to definition of eigenvectors, consider the $i^{th}$ eigenvalues. $X'Xv_i = \lambda_i v_i$

Premultiply $v_i$ with $X'$, we get $(X'X)X'v_i = \lambda_i X'v_i$

We also have $X'X \xi_i = \mu_i \xi_i$. Since $\lambda_i = \mu_i$, we get that $\xi_i = X'v_i$ for $i<p$.

Answer 4

Comparison: "Metric MDS gives the SAME result as PCA"- procedurally- when we look at the way SVD is used to obtain the optimum. But, the preserved high-dimensional criteria is different. PCA uses a centered covariance matrix while MDS uses a gram matrix obtained by double-centering distance matrices.

Will put the difference mathematically: PCA can be viewed as maximizing $Tr(X^T(I-\frac{1}{n}ee^T)X)$ over $X$ under constraints that $X$ is orthogonal, thereby giving axes/principal components. In multidimensional scaling a gram matrix(a p.s.d matrix that can be represented as $Z^TZ$) is computed from euclidean distance between rows in $X$ and the following is minimized over $Y$. minimize: $||G-Y^TY||_{F}^{2}$.