15
$\begingroup$

Are reduced rank regression and principal component regression just special cases of partial least squares?

This tutorial (Page 6, "Comparison of Objectives") states that when we do partial least squares without projecting X or Y (i.e., "not partial"), it becomes reduced rank regression or principal component regression, correspondingly.

A similar statement is made on this SAS documentation page, Sections "Reduced Rank Regression" and "Relationships between Methods".

A more fundamental followup question is whether they have similar underlying probabilistic models.

$\endgroup$
  • $\begingroup$ This is really an important problem. $\endgroup$ – Steve Apr 11 '16 at 5:46
  • $\begingroup$ @Steve. Thanks. See my comments above for a more detailed introduction. $\endgroup$ – Minkov Apr 11 '16 at 22:07
13
$\begingroup$

These are three different methods, and none of them can be seen as a special case of another.

Formally, if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) datasets and if we look for the first pair of axes, $\mathbf w \in \mathbb R^p$ for $\mathbf X$ and $\mathbf v \in \mathbb R^q$ for $\mathbf Y$, then these methods maximize the following quantities:

\begin{align} \mathrm{PCA:}&\quad \operatorname{Var}(\mathbf{Xw}) \\ \mathrm{RRR:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot{}}\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf{Yv}) \\ \mathrm{PLS:}&\quad \operatorname{Var}(\mathbf{Xw})\cdot\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf {Yv}) = \operatorname{Cov}^2(\mathbf{Xw},\mathbf {Yv})\\ \mathrm{CCA:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot {}}\operatorname{Corr}^2(\mathbf {Xw},\mathbf {Yv}) \end{align}

(I added canonical correlation analysis (CCA) to this list.)


I suspect that the confusion might be because in SAS all three methods seem to be implemented via the same function PROC PLS with different parameters. So it might seem that all three methods are special cases of PLS because that's how the SAS function is named. This is, however, just an unfortunate naming. In reality, PLS, RRR, and PCR are three different methods that just happen to be implemented in SAS in one function that for some reason is called PLS.

Both tutorials that you linked to are actually very clear about that. Page 6 of the presentation tutorial states objectives of all three methods and does not say PLS "becomes" RRR or PCR, contrary to what you claimed in your question. Similarly, the SAS documentation explains that three methods are different, giving formulas and intuition:

[P]rincipal components regression selects factors that explain as much predictor variation as possible, reduced rank regression selects factors that explain as much response variation as possible, and partial least squares balances the two objectives, seeking for factors that explain both response and predictor variation.

There is even a figure in the SAS documentation showing a nice toy example where three methods give different solutions. In this toy example there are two predictors $x_1$ and $x_2$ and one response variable $y$. The direction in $X$ that is most correlated with $y$ happens to be orthogonal to the direction of maximal variance in $X$. Hence PC1 is orthogonal to the first RRR axis, and PLS axis is somewhere in between.

PCR, PLS, RRR

One can add a ridge penalty to the RRR lost function obtaining ridge reduced-rank regression, or RRRR. This will pull the regression axis towards the PC1 direction, somewhat similar to what PLS is doing. However, the cost function for RRRR cannot be written in a PLS form, so they remain different.

Note that when there is only one predictor variable $y$, CCA = RRR = usual regression.

$\endgroup$
  • 4
    $\begingroup$ The table at the end is very helpful. Based on that table, one might consider PCA, RRR, and CCA to be "special cases" of PLS if you also think that bicycles and unicycles are special cases of a tricycle. I don't tend to think that way. $\endgroup$ – EdM Apr 12 '16 at 15:19
  • 2
    $\begingroup$ @EdM, I think one can say that all these methods are special cases of some unifying method that does not really have a name (but one can invent it!). But the name "PLS" already has an established meaning and this meaning does not include any of these other techniques. $\endgroup$ – amoeba says Reinstate Monica Apr 12 '16 at 15:25
  • 1
    $\begingroup$ And thanks! I decided now to move the table to the beginning of the answer :) $\endgroup$ – amoeba says Reinstate Monica Apr 12 '16 at 15:28
  • 2
    $\begingroup$ @Moskowitz: Yes, if $X$ is whitened then PLS=RRR and if $Y$ is whitened then RRR=CCA; if both are whitened then PLS=RRR=CCA. However, PCR remains different. Regarding the unifying method, well, one can just say that we maximize $\mathrm{Var}(Xw)^\alpha\cdot \mathrm{Corr}(Xw,Yv)^\beta\cdot \mathrm{Var}(Yv)^\gamma$ and get various methods for various values of alpha, beta, and gamma. Don't think it's very useful though. $\endgroup$ – amoeba says Reinstate Monica Apr 12 '16 at 23:17
  • 1
    $\begingroup$ @Moskowitz: In general, when people talk about method A being a "special case" of method B, they mean that B is more general and A is equivalent to B with some specific parameters. They do not mean that A gives the same results as B under some special conditions on the dataset. Hence my answer to your question. $\endgroup$ – amoeba says Reinstate Monica Apr 12 '16 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.