72
$\begingroup$

Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples on google image search), I have not seen a similar intuitive two-dimensional example for CCA. How to explain visually what linear CCA does?

$\endgroup$
  • 1
    $\begingroup$ In what way CCA generalizes PCA? I wouldn't say it is its generalization. PCA works with one set of variables, CCA works with two (or more, modern implementations), and this is a major difference. $\endgroup$ – ttnphns Jul 27 '13 at 5:11
  • 2
    $\begingroup$ Well, strictly speaking related might be a better choice of word. Anyway, PCA operates on a covariance matrix, and CCA on a cross-covariance matrix. If you have just one dataset, calculating its cross-covariances against itself end up back to the simpler case (PCA). $\endgroup$ – figure Jul 27 '13 at 9:25
  • 4
    $\begingroup$ Well, yes, "related" is better. CCA takes account for both inter-covariances and cross-covariances. $\endgroup$ – ttnphns Jul 27 '13 at 10:48
  • 1
    $\begingroup$ Some have suggested to visualize canonical correlations using heliographs. You might want to read the paper ti.arc.nasa.gov/m/profile/adegani/Composite_Heliographs.pdf $\endgroup$ – user28555 Jul 28 '13 at 14:24
99
+50
$\begingroup$

Well, I think it is really difficult to present a visual explanation of Canonical correlation analysis (CCA) vis-a-vis Principal components analysis (PCA) or Linear regression. The latter two are often explained and compared by means of a 2D or 3D data scatterplots, but I doubt if that is possible with CCA. Below I've drawn pictures which might explain the essence and the differences in the three procedures, but even with these pictures - which are vector representations in the "subject space" - there are problems with capturing CCA adequately. (For algebra/algorithm of canonical correlation analysis look in here.)

Drawing individuals as points in a space where the axes are variables, a usual scatterplot, is a variable space. If you draw the opposite way - variables as points and individuals as axes - that will be a subject space. Drawing the many axes is actually needless because the space has the number of non-redundant dimensions equal to the number of non-collinear variables. Variable points are connected with the origin and form vectors, arrows, spanning the subject space; so here we are (see also). In a subject space, if variables have been centered, the cosine of the angle between their vectors is Pearson correlation between them, and the vectors' lengths squared are their variances. On the pictures below the variables displayed are centered (no need for a constant arises).

Principal Components

enter image description here

Variables $X_1$ and $X_2$ positively correlate: they have acute angle between them. Principal components $P_1$ and $P_2$ lie in the same space "plane X" spanned by the two variables. The components are variables too, only mutually orthogonal (uncorrelated). The direction of $P_1$ is such as to maximize the sum of the two squared loadings of this component; and $P_2$, the remaining component, goes orthogonally to $P_1$ in plane X. The squared lengths of all the four vectors are their variances (the variance of a component is the aforementioned sum of its squared loadings). Component loadings are the coordinates of variables onto the components - $a$'s shown on the left pic. Each variable is the error-free linear combination of the two components, with the corresponding loadings being the regression coefficients. And vice versa, each component is the error-free linear combination of the two variables; the regression coefficients in this combination are given by the skew coordinates of the components onto the variables - $b$'s shown on the right pic. The actual regression coefficient magnitude will be $b$ divided by the product of lengths (standard deviations) of the predicted component and the predictor variable, e.g. $b_{12}/(|P_1|*|X_2|)$. [Footnote: The components' values appearing in the mentioned above two linear combinations are standardized values, st. dev. = 1. This because the information about their variances is captured by the loadings. To speak in terms of unstandardized component values, $a$'s on the pic above should be eigenvectors' values, the rest of the reasoning being the same.]

Multiple Regression

enter image description here

Whereas in PCA everything lies in plane X, in multiple regression there appears a dependent variable $Y$ which usually doesn't belong to plane X, the space of the predictors $X_1$, $X_2$. But $Y$ is perpendicularly projected onto plane X, and the projection $Y'$, the $Y$'s shade, is the prediction by or linear combination of the two $X$'s. On the picture, the squared length of $e$ is the error variance. The cosine between $Y$ and $Y'$ is the multiple correlation coefficient. Like it was with PCA, the regression coefficients are given by the skew coordinates of the prediction ($Y'$) onto the variables - $b$'s. The actual regression coefficient magnitude will be $b$ divided by the length (standard deviation) of the predictor variable, e.g. $b_{2}/|X_2|$.

Canonical Correlation

In PCA, a set of variables predict themselves: they model principal components which in turn model back the variables, you don't leave the space of the predictors and (if you use all the components) the prediction is error-free. In multiple regression, a set of variables predict one extraneous variable and so there is some prediction error. In CCA, the situation is similar to that in regression, but (1) the extraneous variables are multiple, forming a set of their own; (2) the two sets predict each other simultaneously (hence correlation rather than regression); (3) what they predict in each other is rather an extract, a latent variable, than the observed predictand of a regression (see also).

enter image description here

Let's involve the second set of variables $Y_1$ and $Y_2$ to correlate canonically with our $X$'s set. We have spaces - here, planes - X and Y. It should be notified that in order the situation to be nontrivial - like that was above with regression where $Y$ stands out of plane X - planes X and Y must intersect only in one point, the origin. Unfortunately it is impossible to draw on paper because 4D presentation is necessary. Anyway, the grey arrow indicates that the two origins are one point and the only one shared by the two planes. If that is taken, the rest of the picture resembles what was with regression. $V_x$ and $V_y$ are the pair of canonical variates. Each canonical variate is the linear combination of the respective variables, like $Y'$ was. $Y'$ was the orthogonal projection of $Y$ onto plane X. Here $V_x$ is a projection of $V_y$ on plane X and simultaneously $V_y$ is a projection of $V_x$ on plane Y, but they are not orthogonal projections. Instead, they are found (extracted) so as to minimize the angle $\phi$ between them. Cosine of that angle is the canonical correlation. Since projections need not be orthogonal, lengths (hence variances) of the canonical variates are not automatically determined by the fitting algorithm and are subject to conventions/constraints which may differ in different implementations. The number of pairs of canonical variates (and hence the number of canonical correlations) is min(number of $X$s, number of $Y$s). And here comes the time when CCA resembles PCA. In PCA, you skim mutually orthogonal principal components (as if) recursively until all the multivariate variability is exhausted. Similarly, in CCA mutually orthogonal pairs of maximally correlated variates are extracted until all the multivariate variability that can be predicted in the lesser space (lesser set) is up. In our example with $X_1$ $X_2$ vs $Y_1$ $Y_2$ there remains the second and weaker correlated canonical pair $V_{x(2)}$ (orthogonal to $V_x$) and $V_{y(2)}$ (orthogonal to $V_y$).

For the difference between CCA and PCA+regression see also Doing CCA vs. building a dependent variable with PCA and then doing regression.

$\endgroup$
  • 3
    $\begingroup$ +1 (from days ago). I really hope you end up w/ more than 6 upvotes for this; it's a really great overview of how CCA works. $\endgroup$ – gung - Reinstate Monica Jul 31 '13 at 13:12
  • 2
    $\begingroup$ This helps me a lot in understanding CCA. $\endgroup$ – Zhenglei Sep 3 '13 at 11:11
  • $\begingroup$ @Glen_b, I was taken aback, so much pleased that you decided to give award to this answer. $\endgroup$ – ttnphns Sep 4 '13 at 7:36
  • 1
    $\begingroup$ @ttnphns, superb. Even though I did not understand everything, it is definitely by far the best explanation of CCA I have come across. And I think it's really important to get a visual of what's happening, since I know I will remember something if I can visualise it, as opposed to meandering through different theorems. $\endgroup$ – Christian Jan 26 '14 at 12:25
  • $\begingroup$ Do you have a geometric interpretation of how to construct PC1 vector $P_1$ on your first figure given two vectors $X_1$ and $X_2$? You explain how to find its direction and I know how to find its length, but I cannot think of any geometrical interpretation or even intuition of this operation. $\endgroup$ – amoeba says Reinstate Monica Nov 9 '15 at 15:29
2
$\begingroup$

For me it was much helpful to read in the book of S. Mulaik "The Foundations of Factoranalysis" (1972), that there is a method purely of rotations of a matrix of factor loadings to arrive at a canonical correlation, so I could locate it in that ensemble of concepts which I had already understood so far from principal components analysis and factor analysis.

Perhaps you're interested in this example (which I've rebuilt from a first implementation/discussion of about 1998 just a couple of days ago to crosscheck and re-verify the method against the computation by SPSS). See here . I'm using my small matrix/pca-tools Inside-[R] and Matmate for this, but I think it can be reconstructed in R without too much effort.

$\endgroup$
2
$\begingroup$

This answer doesn't provide a visual aid for understanding CCA, however a good geometric interpretation of CCA is presented in Chapter 12 of Anderson-1958 [1]. The gist of it is as follows:

Consider $N$ data points $x_1, x_2, ..., x_N$, all of dimension $p$. Let $X$ be the $p\times N$ matrix containing $x_i$. One way of looking at the data is to interpret $X$ as a collection of $p$ data points in the $(N-1)$-dimensional subspace$^*$. In that case, if we separate the first $p_1$ data points from the remaining $p_2$ data points, CCA tries to find a linear combination of $x_1,...,x_{p_1}$ vectors that is parallel (as parallel as possible) with the linear combination of the remaining $p_2$ vectors $x_{p_1+1}, ..., x_p$.

I find this perspective interesting for these reasons:

  • It provides an interesting geometric interpretation about the entries of CCA canonical variables.
  • The correlation coefficients is linked to the angle between the two CCA projections.
  • The ratios of $\frac{p_1}{N}$ and $\frac{p_2}{N}$ can be directly related to the ability of CCA to find maximally correlated data points. Therefore, the relationship between overfitting and CCA solutions is clear. $\rightarrow$ Hint: The data points are able to span the $(N-1)$-dimensional space, when $N$ is too small (sample-poor case).

Here I've added an example with some code where you can change $p_1$ and $p_2$ and see when they are too high, CCA projections fall on top of each other.

* Note that the sub-space is $(N-1)$-dimensional and not $N$-dimensional, because of the centering constraint (i.e., $\text{mean}(x_i) = 0$).

[1] Anderson, T. W. An introduction to multivariate statistical analysis. Vol. 2. New York: Wiley, 1958.

$\endgroup$
  • 1
    $\begingroup$ Can you add pictures from that book to visualize the answer? $\endgroup$ – ttnphns May 21 '18 at 5:54
  • $\begingroup$ Unfortunately, the book doesn't have pictures for this chapter (in fact I don't think there are any figures in the entire book). $\endgroup$ – idnavid May 21 '18 at 6:05
  • $\begingroup$ @ttnphns I spent some time the other day and put together a small example to illustrate this point. Thanks for the suggestion! $\endgroup$ – idnavid May 31 '18 at 0:30
1
$\begingroup$

The best way to teach statistics is with data. Multivariate statistical techniques are often made very complicated with matrices which are not intuitive. I would explain CCA using Excel. Create two samples, add new variates (columns basically) and show the calculation. And as far as the matrix construction of CCA is concerned, best way is to teach with a bivariate case first and then expand it.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.