20
$\begingroup$

I have been reading The Elements of Statistical Learning and I could not understand what Section 3.7 "Multiple outcome shrinkage and selection" is all about. It talks about RRR (reduced-rank regression), and I can only understand that the premise is about a generalized multivariate linear model where the coefficients are unknown (and is to be estimated) but is known not to have full rank. That's the only thing I understand.

The rest of the mathematics is beyond me. It doesn't even help that the authors say 'one can show' and leaves things as an exercise.

Can someone please help explain what is happening here, intuitively? Is this chapter supposedly discussing new methods? or what?

$\endgroup$
  • 1
    $\begingroup$ It seems to to give regression methods that capitalize on multi outcome models in the context of shrinkage and variable selection. There is not a single Y outcome, but more than one Y outcomes. Let's say you have 5 Y outcomes, then this section discusses methods to pool the estimation of the methods, instead of just building 5 separate models. $\endgroup$ – spdrnl May 15 '15 at 16:12
  • 1
    $\begingroup$ My few cents: assumption of low rank matrix makes things simpler. Fortunately this assumption holds for many real world data sources. $\endgroup$ – Vladislavs Dovgalecs Sep 1 '15 at 19:21
  • 1
    $\begingroup$ It looks like that this assumption is about having the restrictions on the solution. This paper describes why statprob.com/encyclopedia/… $\endgroup$ – Vladislavs Dovgalecs Sep 1 '15 at 19:30
38
$\begingroup$

1. What is reduced-rank regression (RRR)?

Consider multivariate multiple linear regression, i.e. regression with $p$ independent variables and $q$ dependent variables. Let $\mathbf X$ and $\mathbf Y$ be centered predictor ($n \times p$) and response ($n\times q$) datasets. Then usual ordinary least squares (OLS) regression can be formulated as minimizing the following cost function:

$$L=\|\mathbf Y-\mathbf X\mathbf B\|^2,$$

where $\mathbf B$ is a $p\times q$ matrix of regression weights. Its solution is given by $$\hat{\mathbf B}_\mathrm{OLS}=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf Y,$$ and it is easy to see that it is equivalent to doing $q$ separate OLS regressions, one for each dependent variable.

Reduced-rank regression introduces a rank constraint on $\mathbf B$, namely $L$ should be minimized with $\operatorname{rank}(\mathbf B)\le r$, where $r$ is the maximal allowed rank of $\mathbf B$.

2. How to obtain the RRR solution?

It turns out that RRR can be cast as an eigenvector problem. Indeed, using the fact that OLS is essentially orthogonal projection on the column space of $\mathbf X$, we can rewrite $L$ as $$L=\|\mathbf Y-\mathbf X\hat{\mathbf B}_\mathrm{OLS}\|^2+\|\mathbf X\hat{\mathbf B}_\mathrm{OLS}-\mathbf X\mathbf B\|^2.$$ The first term does not depend on $\mathbf B$ and the second term can be minimized by SVD/PCA of the fitted values $\hat{\mathbf Y}=\mathbf X\hat{\mathbf B}_\mathrm{OLS}$.

Specifically, if $\mathbf U_r$ are first $r$ principal axes of $\hat{\mathbf Y}$, then $$\hat{\mathbf B}_\mathrm{RRR}=\hat{\mathbf B}_\mathrm{OLS}\mathbf U_r\mathbf U_r^\top.$$

3. What is RRR good for?

There can be two reasons to use RRR.

First, one can use it for regularization purposes. Similarly to ridge regression (RR), lasso, etc., RRR introduces some "shrinkage" penalty on $\mathbf B$. The optimal rank $r$ can be found via cross-validation. In my experience, RRR easily outperforms OLS but tends to lose to RR. However, RRR+RR can perform (slightly) better than RR alone.

Second, one can use it as a dimensionality reduction / data exploration method. If we have a bunch of predictor variables and a bunch of dependent variables, then RRR will construct "latent factors" in the predictor space that do the best job of explaining the variance of DVs. One can then try to interpret these latent factors, plot them, etc. As far as I know, this is routinely done in ecology where RRR is known as redundancy analysis and is an example of what they call ordination methods (see @GavinSimpson's answer here).

4. Relationship to other dimensionality reduction methods

RRR is closely connected to other dimensionality reduction methods, such as CCA and PLS. I covered it a little bit in my answer to What is the connection between partial least squares, reduced rank regression, and principal component regression?

if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) datasets and if we look for the first pair of axes, $\mathbf w \in \mathbb R^p$ for $\mathbf X$ and $\mathbf v \in \mathbb R^q$ for $\mathbf Y$, then these methods maximize the following quantities:

\begin{align} \mathrm{PCA:}&\quad \operatorname{Var}(\mathbf{Xw}) \\ \mathrm{RRR:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot{}}\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf{Yv}) \\ \mathrm{PLS:}&\quad \operatorname{Var}(\mathbf{Xw})\cdot\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf {Yv}) = \operatorname{Cov}^2(\mathbf{Xw},\mathbf {Yv})\\ \mathrm{CCA:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot {}}\operatorname{Corr}^2(\mathbf {Xw},\mathbf {Yv}) \end{align}

See there for some more details.

See Torre, 2009, A Least-Squares Framework for Component Analysis for a detailed treatment of how most of the common linear multivariate methods (e.g. PCA, CCA, LDA, -- but not PLS!) can be seen as RRR.

5. Why is this section in Hastie et al. so confusing?

Hastie et al. use the term RRR to refer to a slightly different thing! Instead of using the loss function $$L=\|\mathbf Y-\mathbf X \mathbf B\|^2,$$ they use $$L=\|(\mathbf Y-\mathbf X \mathbf B)(\mathbf Y^\top \mathbf Y)^{-1/2}\|^2,$$ as can be seen in their formula 3.68. This introduces a $\mathbf Y$-whitening factor into the loss function, essentially whitening the dependent variables. If you look at the comparison between CCA and RRR above, you will notice that if $\mathbf Y$ is whitened then the difference disappears. So what Hastie et al. call RRR is actually CCA in disguise (and indeed, see their 3.69).

None of that is properly explained in this section, hence the confusion.


See my answer to Friendly tutorial or introduction to reduced-rank regression for further reading.

$\endgroup$
  • $\begingroup$ This is a very nicely written detailed explanation. Thank you, I appreciate it. $\endgroup$ – cgo May 18 '16 at 2:02
  • $\begingroup$ @amoeba Fantastic answer. Can I suggest a couple of retouches to make it more accessible? On the last line of the first entry, can you spell out what $r$ is, for instance, the rank of the model matrix if that is what it is. Second, on the defining equation under the second entry, you introduce $\bf B$, which is the population coefficients, and thus an unknown parameter. Can you elaborate on that a bit? $\endgroup$ – Antoni Parellada May 18 '16 at 9:32
  • 1
    $\begingroup$ (1) It's multivariate regression, @Antoni, i.e. $Y$ is a matrix, and $B$ is a matrix too, not a vector. (2) Here $B$ is just a parameter of the loss-function $L$. The goal is to find $B$ minimizing $L$. $\endgroup$ – amoeba May 18 '16 at 10:19
  • 1
    $\begingroup$ About selecting the optimal rank $r$ in RRRR, note that the degrees of freedom can be explicitly computed as a function of $r$: $\hat{\text{df}}(r) = pq - (p-r)(q-r) + \text{"a small correction term"}$, where $p$ is the input dimension and $q$ is the output dimension. One can then employ generalized cross-validation (GCV) to select the best $r$: minimize $\frac{\|Y - \hat{Y}^{\text{RRRR}}(r)\|_{\text{Fro}}^2}{(nq - \hat{\text{df}}(r))^2}$. $\endgroup$ – dohmatob Apr 12 '17 at 20:28
  • 1
    $\begingroup$ See for example google.fr/url?sa=t&source=web&rct=j&url=https://… $\endgroup$ – dohmatob Apr 12 '17 at 23:49
3
$\begingroup$

Reduced Rank Regression is a model where there is not a single Y outcome, but multiple Y outcomes. Of course, you can just fit a separate multivariate linear regression for each response, but this seems inefficient when the functional relationship between the predictors and each response is clearly similar. See this kaggle exercise for a situation where I believe this obviously holds.

https://www.kaggle.com/c/bike-sharing-demand/data

There are several related techniques for approaching this problem that build "factors" or "components" out of the X variables that are then used for predicting the Ys. This documentation page from SAS helped clear up the differences for me. Reduced Rank Regression seems to be about extracting components that maximally account for variation among the responses, in contrast to Partial Least Squares which extracts components that maximally account for variation among both the responses and predictors.

https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_pls_sect014.htm

$\endgroup$
  • $\begingroup$ +1. That's correct. I discussed this SAS documentation page and in particular their figure in my answer to stats.stackexchange.com/questions/206587. $\endgroup$ – amoeba May 18 '16 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.