How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?

Question

Canonical correlation analysis (CCA) is a technique related to principal component analysis (PCA). While it is easy to teach PCA or linear regression using a scatter plot (see a few thousand examples on google image search), I have not seen a similar intuitive two-dimensional example for CCA. How to explain visually what linear CCA does?

Answer 1

Well, I think it is really difficult to present a visual explanation of Canonical correlation analysis (CCA) vis-a-vis Principal components analysis (PCA) or Linear regression. The latter two are often explained and compared by means of a 2D or 3D data scatterplots, but I doubt if that is possible with CCA. Below I've drawn pictures which might explain the essence and the differences in the three procedures, but even with these pictures - which are vector representations in the "subject space" - there are problems with capturing CCA adequately. (For algebra/algorithm of canonical correlation analysis look in here.)

Drawing individuals as points in a space where the axes are variables, a usual scatterplot, is a variable space. If you draw the opposite way - variables as points and individuals as axes - that will be a subject space. Drawing the many axes is actually needless because the space has the number of non-redundant dimensions equal to the number of non-collinear variables. Variable points are connected with the origin and form vectors, arrows, spanning the subject space; so here we are (see also). In a subject space, if variables have been centered, the cosine of the angle between their vectors is Pearson correlation between them, and the vectors' lengths squared are their variances. On the pictures below the variables displayed are centered (no need for a constant arises).

Principal Components

Variables $X_1$ and $X_2$ positively correlate: they have acute angle between them. Principal components $P_1$ and $P_2$ lie in the same space "plane X" spanned by the two variables. The components are variables too, only mutually orthogonal (uncorrelated). The direction of $P_1$ is such as to maximize the sum of the two squared loadings of this component; and $P_2$, the remaining component, goes orthogonally to $P_1$ in plane X. The squared lengths of all the four vectors are their variances (the variance of a component is the aforementioned sum of its squared loadings). Component loadings are the coordinates of variables onto the components - $a$'s shown on the left pic. Each variable is the error-free linear combination of the two components, with the corresponding loadings being the regression coefficients. And vice versa, each component is the error-free linear combination of the two variables; the regression coefficients in this combination are given by the skew coordinates of the components onto the variables - $b$'s shown on the right pic. The actual regression coefficient magnitude will be $b$ divided by the product of lengths (standard deviations) of the predicted component and the predictor variable, e.g. $b_{12}/(|P_1|*|X_2|)$. [Footnote: The components' values appearing in the mentioned above two linear combinations are standardized values, st. dev. = 1. This because the information about their variances is captured by the loadings. To speak in terms of unstandardized component values, $a$'s on the pic above should be eigenvectors' values, the rest of the reasoning being the same.]

Multiple Regression

Whereas in PCA everything lies in plane X, in multiple regression there appears a dependent variable $Y$ which usually doesn't belong to plane X, the space of the predictors $X_1$, $X_2$. But $Y$ is perpendicularly projected onto plane X, and the projection $Y'$, the $Y$'s shade, is the prediction by or linear combination of the two $X$'s. On the picture, the squared length of $e$ is the error variance. The cosine between $Y$ and $Y'$ is the multiple correlation coefficient. Like it was with PCA, the regression coefficients are given by the skew coordinates of the prediction ($Y'$) onto the variables - $b$'s. The actual regression coefficient magnitude will be $b$ divided by the length (standard deviation) of the predictor variable, e.g. $b_{2}/|X_2|$.

Canonical Correlation

In PCA, a set of variables predict themselves: they model principal components which in turn model back the variables, you don't leave the space of the predictors and (if you use all the components) the prediction is error-free. In multiple regression, a set of variables predict one extraneous variable and so there is some prediction error. In CCA, the situation is similar to that in regression, but (1) the extraneous variables are multiple, forming a set of their own; (2) the two sets predict each other simultaneously (hence correlation rather than regression); (3) what they predict in each other is rather an extract, a latent variable, than the observed predictand of a regression (see also).

Let's involve the second set of variables $Y_1$ and $Y_2$ to correlate canonically with our $X$'s set. We have spaces - here, planes - X and Y. It should be notified that in order the situation to be nontrivial - like that was above with regression where $Y$ stands out of plane X - planes X and Y must intersect only in one point, the origin. Unfortunately it is impossible to draw on paper because 4D presentation is necessary. Anyway, the grey arrow indicates that the two origins are one point and the only one shared by the two planes. If that is taken, the rest of the picture resembles what was with regression. $V_x$ and $V_y$ are the pair of canonical variates. Each canonical variate is the linear combination of the respective variables, like $Y'$ was. $Y'$ was the orthogonal projection of $Y$ onto plane X. Here $V_x$ is a projection of $V_y$ on plane X and simultaneously $V_y$ is a projection of $V_x$ on plane Y, but they are not orthogonal projections. Instead, they are found (extracted) so as to minimize the angle $\phi$ between them. Cosine of that angle is the canonical correlation. Since projections need not be orthogonal, lengths (hence variances) of the canonical variates are not automatically determined by the fitting algorithm and are subject to conventions/constraints which may differ in different implementations. The number of pairs of canonical variates (and hence the number of canonical correlations) is min(number of $X$s, number of $Y$s). And here comes the time when CCA resembles PCA. In PCA, you skim mutually orthogonal principal components (as if) recursively until all the multivariate variability is exhausted. Similarly, in CCA mutually orthogonal pairs of maximally correlated variates are extracted until all the multivariate variability that can be predicted in the lesser space (lesser set) is up. In our example with $X_1$ $X_2$ vs $Y_1$ $Y_2$ there remains the second and weaker correlated canonical pair $V_{x(2)}$ (orthogonal to $V_x$) and $V_{y(2)}$ (orthogonal to $V_y$).

For the difference between CCA and PCA+regression see also Doing CCA vs. building a dependent variable with PCA and then doing regression.

What is the benefit of canonical correlation over individual Pearson correlations of pairs of variables from the two sets? (my answer's in comments).

Answer 2

For me it was much helpful to read in the book of S. Mulaik "The Foundations of Factoranalysis" (1972), that there is a method purely of rotations of a matrix of factor loadings to arrive at a canonical correlation, so I could locate it in that ensemble of concepts which I had already understood so far from principal components analysis and factor analysis.

Perhaps you're interested in this example (which I've rebuilt from a first implementation/discussion of about 1998 just a couple of days ago to crosscheck and re-verify the method against the computation by SPSS). See here . I'm using my small matrix/pca-tools Inside-[R] and Matmate for this, but I think it can be reconstructed in R without too much effort.

Answer 3

This answer doesn't provide a visual aid for understanding CCA, however a good geometric interpretation of CCA is presented in Chapter 12 of Anderson-1958 [1]. The gist of it is as follows:

Consider $N$ data points $x_1, x_2, ..., x_N$, all of dimension $p$. Let $X$ be the $p\times N$ matrix containing $x_i$. One way of looking at the data is to interpret $X$ as a collection of $p$ data points in the $(N-1)$-dimensional subspace$^*$. In that case, if we separate the first $p_1$ data points from the remaining $p_2$ data points, CCA tries to find a linear combination of $x_1,...,x_{p_1}$ vectors that is parallel (as parallel as possible) with the linear combination of the remaining $p_2$ vectors $x_{p_1+1}, ..., x_p$.

I find this perspective interesting for these reasons:

• It provides an interesting geometric interpretation about the entries of CCA canonical variables.
• The correlation coefficients is linked to the angle between the two CCA projections.
• The ratios of $\frac{p_1}{N}$ and $\frac{p_2}{N}$ can be directly related to the ability of CCA to find maximally correlated data points. Therefore, the relationship between overfitting and CCA solutions is clear. $\rightarrow$ Hint: The data points are able to span the $(N-1)$-dimensional space, when $N$ is too small (sample-poor case).

Here I've added an example with some code where you can change $p_1$ and $p_2$ and see when they are too high, CCA projections fall on top of each other.

* Note that the sub-space is $(N-1)$-dimensional and not $N$-dimensional, because of the centering constraint (i.e., $\text{mean}(x_i) = 0$).

[1] Anderson, T. W. An introduction to multivariate statistical analysis. Vol. 2. New York: Wiley, 1958.

Answer 4

The best way to teach statistics is with data. Multivariate statistical techniques are often made very complicated with matrices which are not intuitive. I would explain CCA using Excel. Create two samples, add new variates (columns basically) and show the calculation. And as far as the matrix construction of CCA is concerned, best way is to teach with a bivariate case first and then expand it.