It is well known that the solution to the optimization problems proposed in Principal Components and Canonical Correlation Analysis are given by the solution to eigenvalue problems and generalized eigenvalue problems respectively. My linear algebra is quite rusty and I wanted to know if anyone could give me a semi-formal, albeit intuitive explanation on why this is so.

  • 2
    $\begingroup$ Step-by-step math algorithm of CCA is given here. Algorithm of PCA is easier and is repeated in numerous places on this site; to cite some of my own posts: decomposition; loadings; loadings vs eigenvectors; complete output. $\endgroup$ – ttnphns Mar 3 '14 at 6:45
  • $\begingroup$ The problem of CCA is a bit closer to multivariate multiple regression than to PCA: 1, 2. $\endgroup$ – ttnphns Mar 3 '14 at 6:46
  • $\begingroup$ If your question is specifically about eigen-decompositions... In PCA, we have one set of variables, and so the matrix to decompose is symmetric (=> standard eigen-problem). In CCA or discriminant analysis we have 2 sets counter each other, so the matrix is asymmetric (=> generalized eigen-problem, it can be solved in various bypassing tricks). $\endgroup$ – ttnphns Mar 3 '14 at 6:57
  • 3
    $\begingroup$ I recommend this simple and concise paper: M. Borga, T. Landelius, H. Knutsson (1998). A Unified Approach to PCA, PLS, MLR and CCA. $\endgroup$ – user603 Mar 3 '14 at 10:11
  • 1
    $\begingroup$ Perhaps my CCA by rotation is instructive for an old rusty linear-algebra dog... See go.helms-net.de/stat/sse/cancorr150712.htm $\endgroup$ – Gottfried Helms Jul 13 '15 at 16:50