Multiple dim responding variable the relation between CCA and trivial linear regression (PLSR, CCA, PCA, PCR and Linear Regression)

Question

Here is my summary of Multivariate Linear Regression between explain variable $\textbf{x}$ and responding variable $\textbf{y}.$ I have summaried the relation between PLSR, CCA, PCA, PCR and Linear Regression.

However I still left one relation of the CCA and trivial regression which means that regress each column of $\textbf{y}$ respect to the sample matrix of $\textbf{x}$ and simply aggregates the coefficients (widely used in NIPLAS). Since I don't know how to implement the regression of CCA, I guess above two things are essentially equivalent?

Summary:

1. $$\textbf{PLSR:} \max\limits_{||\alpha_y|| = |\alpha_x|| = 1} Cov(\alpha_x\cdot\textbf{x},\ \alpha_y\cdot\textbf{y}).$$

   PLSR simultaneously considers the variance of $\textbf{x}$, variance of $\textbf{y}$ and correlation between $\textbf{x}$ and $\textbf{y}$

   Special case $\dim \textbf{y} = 1: \textbf{PCR}.$

2. $$\textbf{CCA:} \max\limits_{\alpha_x,\alpha_y}\rho(\alpha_x\cdot\textbf{x},\ \alpha_y\cdot\textbf{y})$$ $$\Leftrightarrow$$ $$\max\limits_{\alpha_x,\alpha_y}Cov(\alpha_x\cdot\textbf{x},\ \alpha_y\cdot\textbf{y})$$ $$s.t.\ Var(\alpha_x\cdot\textbf{x}) = Var(\alpha_y\cdot\textbf{y}) = 1.$$ CCA only consider the correlation between $\textbf{x}$ and $\textbf{y}.$

   Special case $\dim \textbf{y} = 1: \textbf{Linear regression}$

3. $$\textbf{PCA:} \max\limits_{||\alpha_x|| = 1} Cov(\alpha_x\cdot\textbf{x},\ \alpha_x\cdot\textbf{x}).$$ PCA only consider the variance of $\textbf{x}$.

$$PCR = X \xrightarrow{PCA} Y \xrightarrow{MLR} \textbf{y}.$$

Answer 1

Canonical correlation analysis (CCA) is mainly concerned with characterizing the linear association between two blocks of variables (or more blocks in generalized CCA). We can think of it as an extension of the use of a correlation matrix to summarize correlations in a multivariate dataset. In this respect, no single block plays the role of a response block which might motivates the idea of regressing one (or more) block onto another one. Partial least squares approaches are better suited for that purpose.

However, there already is a very instructive thread on this site: How to visualize what canonical correlation analysis does (in comparison to what principal component analysis does)?, which also points to another thread related to CCA versus PCA+regression.

Finally, I should note that there's now a unified approach to all such multi-blocks approaches ((generalized) CCA, interbattery factor analysis, redundancy analysis, hierarchical PCA , multiple co-inertia analysis and PLS path modeling), the Regularized Generalized Canonical Correlation Analysis, available in R, theorized and implemented by father and son. Original reference and extension below:

• Tenenhaus, A., and M. Tenenhaus. 2011. Regularized Generalized Canonical Correlation Analysis. Psychometrika, 76: 257–84.
• Tenenhaus, Arthur, Cathy Philippe, and Vincent Frouin. 2015. Kernel Generalized Canonical Correlation Analysis. Computational Statistics & Data Analysis, 90: 114–31.