Reference for dimension reduction techniques

Question

This is a follow-up question to Is PCA appropriate for comparing subsets of panel data?.

It turns out that, yes, PCA is appropriate. But there are also many other ways to reduce n-dimensional data to 1-3 dimensions that can be visualized, and I'm having trouble getting a global view of the available set of techniques for doing so. For instance, several are mentioned in this question and this question. The latter question has a link to Nonlinear Dimensionality Reduction on Wikipedia, which is relatively comprehensive, even encyclopedic, for a Wikipedia article on a technical subject. But it's still Wikipedia and I'd like to have everything in one place. edit: there's also this question from today.

Is there a somewhat-comprehensive reference for dimension reduction? For instance, I'd really like an explanation of why the "manifold-based" approaches (e.g. ISOMAP) are different or better than distance-based approaches (e.g. classical MDS). Ideally it would include some treatment of different distance metrics as well. Ultimately I'm trying to get a handle on the kinds of questions each technique is designed to answer.

Obviously this could turn out to be a bigger subject than I realize. I'm aware that, e.g., self-organizing maps are a type of neural network, and neural networks are quite different "under the hood" from, say, multidimensional scaling. I'm hoping that a reference or set of references exists that doesn't require me to know the gory underlying details of each approach. Learning everything there is to know about statistics is a long process and I'm not quite there yet (although I'm making good progress, in part thanks to this site).

Answer 1

Here's a good paper comparing various dimension reduction techniques to PCA: http://www.iai.uni-bonn.de/~jz/dimensionality_reduction_a_comparative_review.pdf

In brief, the paper covers the following techniques, though there are many more:

(1) multidimensional scaling, (2) Isomap, (3) Maximum Variance Unfolding, (4) Kernel PCA, (5) diffusion maps, (6) multilayer autoencoders, (7) Locally Linear Embedding, (8) Laplacian Eigenmaps, (9) Hessian LLE, (10) Local Tangent Space Analysis, (11) Locally Linear Coordination, and (12) manifold charting

In fact, many of the methods not covered are briefly described in Appendix A.

Moving into the paper, you can see that those above techniques are described in some detail as to their theoretical underpinnings. Table 1 compares the algorithms on convexity of optimization, parameters, computational complexity, and memory complexity.

Table 4 shows the results of the algorithms being run on many example datasets, and the resulting generalization error rates.

In the end, the authors believe that PCA is quite useful/good. For more info on dimension reduction, see his references.

Edit: here's a specific sentence highlighting what you wanted to know about MDS vs Isomap:

Performing MDS using geodesic distances is identical to performing Isomap.

Bonus: My favorite paper using PCA analysis