I have learnt (from another answer) that

CCA will find pairs of vectors ($w$,$v$) such that projections $X_w$ and $Y_v$ have maximal possible correlations (the pairs will be ordered in the order of decreasing correlations). Projection vectors are normalized such that the variance of $X_w$ and of $Y_v$ is equal to $1$. This means that projections are not only correlated, but "on the same scale" and hence can be directly compared.

So if I plot $w$ vs $v$ (the components of the CCA), should all the samples be on the diagonal $w = v$ or $w = -v$?

From what I learned, they should be on the diagonal, but on some plots I have samples that are not perfectly fit on the diagonal and I don't understand why.


1 Answer 1


They are not necessarily confined to the diagonal. The underlying assumption behind CCA is that $X$ and $Y$ share some low-dimensional latent factor, so that $X \approx Az_x + Cz$ and $Y \approx Bz_y + Dz$ with $z_x, z_y, z$ all independent. CCA approximates the shared latent factor $z$ from both ends by trying to find projectors $w$ and $v$ that invert $C$ and $D$ (and map the column spaces of A or B to 0). But, it's not guaranteed to succeed perfectly, nor are the data guaranteed to conform to that generative model.


If you believe the model, then the distance between the samples and the diagonal would measure both contamination by noise (implied by the $\approx$) and the variation due to $z_x$ or $z_y$. For example, if $A=B=0$ and $C$, $D$ are $\begin{bmatrix} 1 &1 \\ 1 & 0 \\ 1 & 0 \end{bmatrix}$, and if the $\approx$ is taken to mean that iid Gaussian noise gets added to the entries after everything is finished, then CCA might use the following projections. (Caveat: I haven't done the actual math to confirm these are correct, and they also are not normalized).

  • $[0, 1, 1 ]$ (reconstructs the first latent coordinate)
  • $[2, -1, -1]$ (reconstructs the second latent coordinate)

But even though I've given CCA the oracular truth about the projections it should be using, the noise means that those projection cannot exactly reconstruct the latent variates as the occurred during the data generation. If we make $A$ and $B$ nonzero, then that affects the results as well.

Worst-case scenario

In the situation that $A=B=C=D$, the projections are not even reconstructing $z_i$ for case $i$. They are at best recovering $z_{x,i}+ z_i$ and $z_{y,i} + z_i$. If $A=500C$, then this worsens to $500z_{x,i} + z_i$. This is no longer a problem with estimation error so much as a fundamental identifiability issue. Even infinite data won't help. If $A$ and $B$ are merely similar to $C$ and $D$, that will also make things really difficult, unless you have an infinite amount of data.

  • $\begingroup$ So if the samples are not on the diagonal there is a need for other variables to explain them? Then the distance betweeb the samples and the diagonal would be an indication of the fitness/correctness of the model, is it? $\endgroup$
    – llrs
    Commented Oct 22, 2017 at 20:02
  • $\begingroup$ Edited answer to add an example; hopefully that'll help. $\endgroup$ Commented Oct 22, 2017 at 21:52
  • $\begingroup$ The example is helpful, but I don't understand the worst-case scenario. I do understand that the effect size of a variable has to be taken into account and that too large effect size won't help but I lost the reason where $A=B=C=D$ is not reconstructing $z_i$. I will wait a bit to mark your answer (if no other answer appears). $\endgroup$
    – llrs
    Commented Oct 23, 2017 at 5:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.