What is the advantage of reducing dimensionality of predictors for the purposes of regression? What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any dimensionality reduction)? These class of techniques finds a low-dimensional representation of the feature set for the regression problem. Examples of such techniques include Sliced Inverse Regression, Principal Hessian Directions, Sliced Average Variance Estimation, Kernel Sliced Inverse Regression, Principal Components Regression, etc.


*

*In terms of cross-validated RMSE, if an algorithm performed better on a regression task without any dimensionality reduction, then what is the real use of the dimensionality reduction for regression? I don't get the point of these techniques.

*Are these techniques by any chance used to reduce space and time complexity for regression? If that is the primary advantage, some resources on complexity reduction for high-dimensional datasets when this techniques are put to use would be helpful. I debate this with the fact that running a DRR or SDR technique itself requires some time and space. Is this SDR/DRR + Regression on a low-dim dataset faster than only regression on a high-dim dataset? 

*Has this setting been studied out of abstract interest only, and does not have a good practical application?
As a side thought: at times there are assumptions that the joint distribution of the features $X$ and the response $Y$ lies on a manifold. It makes sense to learn the manifold from the observed sample in this context for solving a regression problem.
 A: The purpose of dimensionality reduction in regression is regularization.
Most of the techniques that you listed are not very well known; I have not heard about any of them apart from principal components regression (PCR). So I will reply about PCR but expect that the same applies to the other techniques as well.
The two key words here are overfitting and regularization. For a long treatment and discussion I refer you to  The Elements of Statistical Learning, but very briefly, what happens if you have a lot of predictors ($p$) and not enough samples ($n$) is that standard regression will overfit the data and you will construct a model that seems to have good performance on the training set but actually has very poor performance on any test set.
In an extreme example, when number of predictors exceeds number of samples (people refer to it as to $p>n$ problem), you can actually perfectly fit any response variable $y$, achieving seemingly $100\%$ performance. This is clearly nonsense.
To deal with overfitting one has to use regularization, and there are plenty of different regularization strategies. In some approaches one tries to drastically reduce the number of predictors, reducing the problem to the $p\ll n$ situation, and then to use standard regression. This is exactly what principal components regression does. Please see The Elements, sections 3.4--3.6. PCR is usually suboptimal and in most cases some other regularization methods will perform better, but it is easy to understand and interpret.
Note that PCR is not arbitrary either (e.g. randomly keeping $p$ dimensions is likely to perform much worse). The reason for this is that PCR is closely connected to ridge regression, which is a standard shrinkage regularizer that is known to work well in a large variety of cases. See my answer here for the comparison: Relationship between ridge regression and PCA regression.
To see a performance increase compared to standard regression, you need a dataset with a lot of predictors and not so many samples, and you definitely need to use cross-validation or an independent test set. If  you did not see any performance increase, then perhaps your dataset did not have enough dimensions.
Related threads with good answers:


*

*Regression in $p\gg N$ setting (predicting drug efficiency from gene expression with 30k predictors and ~30 samples)

*Regression in $p>n$ setting: how to choose regularization method (Lasso, PLS, PCR, ridge)?
A: According to the manifold hypothesis, the data is assumed to lie on a low-dimensional manifold, the implication being that the residual is noise, so if you do your dimensionality reduction correctly, you should improve performance by modeling the signal rather than the noise. It's not just a question of space and complexity.
