This answer doesn't provide a visual aid for understanding CCA, however a good geometric interpretation of CCA is presented in Chapter 12 of Anderson-1958 [1]. The gist of it is as follows:
Consider N$N$ data points x_1, x_2, ..., x_N$x_1, x_2, ..., x_N$, all of dimension p$p$. Let X$X$ be the pxN$p\times N$ matrix containing x_i$x_i$. One way of looking at the data is to interpret X$X$ as a collection of p$p$ data points in the (N-1)$(N-1)$-dimensional subspace*subspace$^*$. In that case, if we separate the first p1$p_1$ data points from the remaining p2$p_2$ data points, CCA tries to find a linear combination of x_1,...,x_p1$x_1,...,x_{p_1}$ vectors that is parallel (as parallel as possible) with the linear combination of the remaining p2$p_2$ vectors x_{p1+1}, ..., x_p$x_{p_1+1}, ..., x_p$.
I find this perspective interesting for these reasons:
- It provides an interesting geometric interpretation about the entries of CCA canonical variables.
- The correlation coefficients is linked to the angle between the two CCA projections.
- The ratios of p1/N$\frac{p_1}{N}$ and p2/N$\frac{p_2}{N}$ can be directly related to the ability of CCA to find maximally correlated data points. Therefore, the relationship between overfitting and CCA solutions is clear. $\rightarrow$ Hint: The data points are able to span the (N-1)$(N-1)$-dimensional space, when N$N$ is too small (sample-poor case).
Here I've added an example with some code where you can change p_1$p_1$ and p_2$p_2$ and see when they are too high, CCA projections fall on top of each other.
* Note that the sub-space is (N-1)$(N-1)$-dimensional and not N$N$-dimensional, because of the centering constraint (i.e., mean(x_i) = 0$\text{mean}(x_i) = 0$).
[1] Anderson, T. W. An introduction to multivariate statistical analysis. Vol. 2. New York: Wiley, 1958.
