How are PCA, LDA, CCA, and PLS related? They all seem "spectral" and linear algebraic and very well understood (say 50+ years of theory built around them). They are used for very different things (PCA for dimensionality reduction, LDA for classification, PLS for regression) but still they feel very closely related.
$\begingroup$
$\endgroup$
-
3$\begingroup$ In addition to the nice reference in the answer below, you can also read Borga et al., 1997, A Unified Approach to PCA, PLS, MLR and CCA. $\endgroup$ – amoeba Jan 30 '15 at 17:43
$\begingroup$
$\endgroup$
Tijl De Bie wrote an interesting chapter "Eigenproblems in Pattern Recognition" which talks about exactly these from a primal/dual perspective. The three tables at the end summarise really nicely from an optimisation perspective:
