1. What is reduced-rank regression (RRR)?
Consider multivariate multiple linear regression, i.e. regression with $p$ independent variables and $q$ dependent variables. Let $\mathbf X$ and $\mathbf Y$ be centered predictor ($n \times p$) and response ($n\times q$) datasets. Then usual ordinary least squares (OLS) regression can be formulated as minimizing the following cost function:
$$L=\|\mathbf Y-\mathbf X\mathbf B\|^2,$$
where $\mathbf B$ is a $p\times q$ matrix of regression weights. Its solution is given by $$\hat{\mathbf B}_\mathrm{OLS}=(\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf Y,$$ and it is easy to see that it is equivalent to doing $q$ separate OLS regressions, one for each dependent variable.
Reduced-rank regression introduces a rank constraint on $\mathbf B$, namely $L$ should be minimized with $\operatorname{rank}(\mathbf B)\le r$, where $r$ is the maximal allowed rank of $\mathbf B$.
2. How to obtain the RRR solution?
It turns out that RRR can be cast as an eigenvector problem. Indeed, using the fact that OLS is essentially orthogonal projection on the column space of $\mathbf X$, we can rewrite $L$ as $$L=\|\mathbf Y-\mathbf X\hat{\mathbf B}_\mathrm{OLS}\|^2+\|\mathbf X\hat{\mathbf B}_\mathrm{OLS}-\mathbf X\mathbf B\|^2.$$ The first term does not depend on $\mathbf B$ and the second term can be minimized by SVD/PCA of the fitted values $\hat{\mathbf Y}=\mathbf X\hat{\mathbf B}_\mathrm{OLS}$.
Specifically, if $\mathbf U_r$ are first $r$ principal axes of $\hat{\mathbf Y}$, then $$\hat{\mathbf B}_\mathrm{RRR}=\hat{\mathbf B}_\mathrm{OLS}\mathbf U_r\mathbf U_r^\top.$$
3. What is RRR good for?
There can be two reasons to use RRR.
First, one can use it for regularization purposes. Similarly to ridge regression (RR), lasso, etc., RRR introduces some "shrinkage" penalty on $\mathbf B$. The optimal rank $r$ can be found via cross-validation. In my experience, RRR easily outperforms OLS but tends to lose to RR. However, RRR+RR can perform (slightly) better than RR alone.
Second, one can use it as a dimensionality reduction / data exploration method. If we have a bunch of predictor variables and a bunch of dependent variables, then RRR will construct "latent factors" in the predictor space that do the best job of explaining the variance of DVs. One can then try to interpret these latent factors, plot them, etc. As far as I know, this is routinely done in ecology where RRR is known as redundancy analysis and is an example of what they call ordination methods (see @GavinSimpson's answer here).
4. Relationship to other dimensionality reduction methods
RRR is closely connected to other dimensionality reduction methods, such as CCA and PLS. I covered it a little bit in my answer to What is the connection between partial least squares, reduced rank regression, and principal component regression?
if $\mathbf X$ and $\mathbf Y$ are centered predictor ($n \times p$) and response ($n\times q$) datasets and if we look for the first pair of axes, $\mathbf w \in \mathbb R^p$ for $\mathbf X$ and $\mathbf v \in \mathbb R^q$ for $\mathbf Y$, then these methods maximize the following quantities:
\begin{align}
\mathrm{PCA:}&\quad \operatorname{Var}(\mathbf{Xw}) \\
\mathrm{RRR:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot{}}\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf{Yv}) \\
\mathrm{PLS:}&\quad \operatorname{Var}(\mathbf{Xw})\cdot\operatorname{Corr}^2(\mathbf{Xw},\mathbf {Yv})\cdot\operatorname{Var}(\mathbf {Yv}) = \operatorname{Cov}^2(\mathbf{Xw},\mathbf {Yv})\\
\mathrm{CCA:}&\quad \phantom{\operatorname{Var}(\mathbf {Xw})\cdot {}}\operatorname{Corr}^2(\mathbf {Xw},\mathbf {Yv})
\end{align}
See there for some more details.
See Torre, 2009, A Least-Squares Framework for Component Analysis for a detailed treatment of how most of the common linear multivariate methods (e.g. PCA, CCA, LDA, -- but not PLS!) can be seen as RRR.
5. Why is this section in Hastie et al. so confusing?
Hastie et al. use the term RRR to refer to a slightly different thing! Instead of using the loss function $$L=\|\mathbf Y-\mathbf X \mathbf B\|^2,$$ they use $$L=\|(\mathbf Y-\mathbf X \mathbf B)(\mathbf Y^\top \mathbf Y)^{-1/2}\|^2,$$ as can be seen in their formula 3.68. This introduces a $\mathbf Y$-whitening factor into the loss function, essentially whitening the dependent variables. If you look at the comparison between CCA and RRR above, you will notice that if $\mathbf Y$ is whitened then the difference disappears. So what Hastie et al. call RRR is actually CCA in disguise (and indeed, see their 3.69).
None of that is properly explained in this section, hence the confusion.
See my answer to Friendly tutorial or introduction to reduced-rank regression for further reading.
