According to Wikipedia, canonical correlation analysis (CCA) finds pairs of canonical variables. CCA has also been used in many cases as dimensionality reduction tool to find low-dimensional subspaces. I am wondering how the subspace is found? and how the subspace is related to the pairs of canonical variables?

  • $\begingroup$ Related or even possible duplicate: stats.stackexchange.com/q/65692/3277 $\endgroup$ – ttnphns Jan 15 '15 at 5:05
  • $\begingroup$ Are you asking to explain mathematically how CCA works? Or are you asking about intuition behind CCA? Or are you asking specifically about the connection between canonical variables and the subspace? This last connection is like the connection between principal components and principal axes (eigenvectors of the covariance matrix) in PCA. $\endgroup$ – amoeba says Reinstate Monica Jan 15 '15 at 9:59
  • $\begingroup$ If this is of interest: I've done a demonstration how the canonical correlations are computed from correlations resp. from the factor-loadings matrix, and have even added a short discussion when the varimax-rotation might even be superior over the usual principal axes-solution. It is a small (old) "living letter" to a friend using my 1996 Dos-program "Inside-R" without deeper introduction. One downloads the zip-file, unzips it in a directory where a Dos-process can work on, opens a Dos-Box and starts the demo. See go.helms-net.de/stat/ir/cc.zip If you have questions, ask here $\endgroup$ – Gottfried Helms Jan 15 '15 at 11:03
  • $\begingroup$ @amoeba, I am asking for the connection between canonical variables and the subspace, is it simply the canonical variables spans the subspace? If I have n pairs of variables, no orthogonality within the pair, but orthogonal between the pairs, the spanned subspace is still dimension n? $\endgroup$ – fast tooth Jan 15 '15 at 15:00
  • $\begingroup$ Say you have two datasets with $n$ points each, $X$ with $p$ dimensions and $Y$ with $q$ dimensions. CCA will find you $m=\min(p,q)$ canonical pairs; this means it will find $m$ canonical axes in the $p$-dimensional $X$ space and $m$ canonical axes in the $q$-dimensional $Y$ space such that projections of the data onto each pair of these axes (called a pair of canonical variables) are maximally correlated. If you take e.g. first 2 pairs only, then first two axes span a 2d subspace in space $X$ and a 2d subspace in space $Y$. Does this make sense? $\endgroup$ – amoeba says Reinstate Monica Jan 15 '15 at 15:08

This question was based on a false premise that CCA finds one "common subspace". It does not.

CCA deals with two datasets $X$ and $Y$ of $n$ points each: points from dataset $X$ are $p$-dimensional and live in $\mathbb R^p$ and points from dataset $Y$ are $q$-dimensional and live in $\mathbb R^q$. Let $\mathbf X$ and $\mathbf Y$ be two centered data matrices of $n\times p$ and $n\times q$ size respectively.

CCA finds $m=\min(p,q)$ pairs of canonical axes. The first pair $(\mathbf w_1, \mathbf v_1)$ consists of one canonical axis $\mathbf w_1 \in \mathbb R^p$ and one canonical axis $\mathbf v_1 \in \mathbb R^p$. Projections of the data onto these axes (called "canonical components", "canonical variates", or "canonical variables") are given by $\mathbf X \mathbf w_1$ and $\mathbf Y \mathbf v_1$, and they have highest possible correlation between each other. Projections of the data on the next pair, $\mathbf w_2$ and $\mathbf v_2$ have second highest correlation, etc.

So the first pair of canonical axes defines a 1-dimensional subspace in each space, but these are two different subspaces in two different spaces. Two first pairs define a 2-dimensional subspace, but these are again two different subspaces. There is never a "common subspace", because the spaces $X$ and $Y$ are different to begin with.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.