Reading Wikipedia about canonical correlation analysis (CCA) for two random vectors $X$ and $Y$, I was wondering if principal component anslysis (PCA) is the same as CCA when $X=Y$?

  • $\begingroup$ Please make it more clear: 1) vectors X and Y Is that two variables (columns of data) or two cases (rows); given that we're going to perform the analyses of variables. 2) X and Y are the same Did you want to say that X=Y or any other way round? $\endgroup$ – ttnphns Jun 15 '13 at 7:25
  • $\begingroup$ @ttnphns: 1) $X$ and $Y$ are two random vectors. They are two vectors of random variables, two sets of columns of data, not two cases (rows). 2) $X=Y$. $\endgroup$ – Tim Jun 15 '13 at 12:01
  • $\begingroup$ If each set consists of a single variable, there's one canonical correlation which is exactly the Pearson r between them; and CCA becomes linear regression of X by Y and vice versa. Decomposition of that r by means of PCA is a bit another story. PCA and CCA are different analyses. $\endgroup$ – ttnphns Jun 15 '13 at 15:18
  • $\begingroup$ Hi, @Tim, I am wondering if my answer was useful or if you maybe still have some further questions? If so, I would be happy to clarify. $\endgroup$ – amoeba says Reinstate Monica Dec 11 '14 at 0:39
  • $\begingroup$ @amoeba: Yes, it is. I don't have further questions right now, and will read your reply later. Thanks for your reply .+1 $\endgroup$ – Tim Dec 11 '14 at 2:57

Let $\bf X$ be $n\times p_1$ and $\bf Y$ be $n \times p_2$ data matrices, representing two datasets with $n$ samples (i.e. observations of your random row vectors $X$ and $Y$) in each of them.

CCA looks for a linear combination of $p_1$ variables in $\bf X$ and a linear combination of $p_2$ variables in $\bf Y$ such that they are maximally correlated between each other; then it looks for the next pair, under a constraint of zero correlation with the first pair; etc.

In case $\mathbf{X} = \mathbf{Y}$ (and $p_1=p_2 = p$), any linear combination in one dataset will trivially have correlation $1$ with the same linear combination in another dataset. So all CCA pairs will have correlations $1$, and the order of pairs is arbitrary. The only remaining constraint is that linear combinations should be uncorrelated between each other. There is an infinite number of ways to choose $p$ uncorrelated linear combinations (note that the weights do not have to be orthogonal in the $p$-dimensional space) and any of them will produce a valid CCA solution. One such way is indeed given by PCA, as any two PCs have correlation zero.

So PCA solution will indeed be a valid CCA solution, but there is an infinite number of equivalently good CCA solutions in this case.

Mathematically, CCA looks for right ($\mathbf a$) and left ($\mathbf b$) singular vectors of $\mathbf C_{XX}^{-1/2} \mathbf C_{XY} \mathbf C_{YY}^{-1/2}$, which in this case is equal to $\mathbf I$, with any vector being an eigenvector. So $\mathbf a=\mathbf b$ can be arbitrary. CCA then obtains the linear combination weights as $\mathbf C_{XX}^{-1/2} \mathbf a$ and $\mathbf C_{YY}^{-1/2} \mathbf b$. In this case it boils down to taking an arbitrary basis and transforming it with $\mathbf C_{XX}^{-1/2}$, which will indeed produce uncorrelated directions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.