What is the advantage of reducing dimensionality of predictors for the purposes of regression?

What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any dimensionality reduction)? These class of techniques finds a low-dimensional representation of the feature set for the regression problem. Examples of such techniques include Sliced Inverse Regression, Principal Hessian Directions, Sliced Average Variance Estimation, Kernel Sliced Inverse Regression, Principal Components Regression, etc.

1. In terms of cross-validated RMSE, if an algorithm performed better on a regression task without any dimensionality reduction, then what is the real use of the dimensionality reduction for regression? I don't get the point of these techniques.

2. Are these techniques by any chance used to reduce space and time complexity for regression? If that is the primary advantage, some resources on complexity reduction for high-dimensional datasets when this techniques are put to use would be helpful. I debate this with the fact that running a DRR or SDR technique itself requires some time and space. Is this SDR/DRR + Regression on a low-dim dataset faster than only regression on a high-dim dataset?

3. Has this setting been studied out of abstract interest only, and does not have a good practical application?

As a side thought: at times there are assumptions that the joint distribution of the features $X$ and the response $Y$ lies on a manifold. It makes sense to learn the manifold from the observed sample in this context for solving a regression problem.

According to the manifold hypothesis, the data is assumed to lie on a low-dimensional manifold, the implication being that the residual is noise, so if you do your dimensionality reduction correctly, you should improve performance by modeling the signal rather than the noise. It's not just a question of space and complexity.

• but I do not see techniques like SIR doing better after dimensionality reduction on a robust basis. Correct me if am wrong or if you know of a SDR/DDR technique that can find this signal better-in a regression setting, let me know of what technique (name) it is. – hearse Dec 10 '14 at 23:46
• Of course it depends on the regression algorithm, and the intrinsic dimensionality of the data. I can't speak for SIR in particular, but here's a paper that compares various regression algorithms on the MNIST dataset, which is low dimensional. Maybe you could share some troublesome data so people can take a crack at it. – Emre Dec 11 '14 at 0:13
• What is "the manifold hypothesis"? – amoeba says Reinstate Monica Dec 11 '14 at 9:24
• – Emre Dec 11 '14 at 9:27
• I wonder if this stuff is similar to neural networks and nonlinear multidimensional scaling in that it "sounds like" it should be great everywhere but in practice does well in a more limited set of cases – shadowtalker Dec 11 '14 at 11:28

The purpose of dimensionality reduction in regression is regularization.

Most of the techniques that you listed are not very well known; I have not heard about any of them apart from principal components regression (PCR). So I will reply about PCR but expect that the same applies to the other techniques as well.

The two key words here are overfitting and regularization. For a long treatment and discussion I refer you to The Elements of Statistical Learning, but very briefly, what happens if you have a lot of predictors ($p$) and not enough samples ($n$) is that standard regression will overfit the data and you will construct a model that seems to have good performance on the training set but actually has very poor performance on any test set.

In an extreme example, when number of predictors exceeds number of samples (people refer to it as to $p>n$ problem), you can actually perfectly fit any response variable $y$, achieving seemingly $100\%$ performance. This is clearly nonsense.

To deal with overfitting one has to use regularization, and there are plenty of different regularization strategies. In some approaches one tries to drastically reduce the number of predictors, reducing the problem to the $p\ll n$ situation, and then to use standard regression. This is exactly what principal components regression does. Please see The Elements, sections 3.4--3.6. PCR is usually suboptimal and in most cases some other regularization methods will perform better, but it is easy to understand and interpret.

Note that PCR is not arbitrary either (e.g. randomly keeping $p$ dimensions is likely to perform much worse). The reason for this is that PCR is closely connected to ridge regression, which is a standard shrinkage regularizer that is known to work well in a large variety of cases. See my answer here for the comparison: Relationship between ridge regression and PCA regression.

To see a performance increase compared to standard regression, you need a dataset with a lot of predictors and not so many samples, and you definitely need to use cross-validation or an independent test set. If you did not see any performance increase, then perhaps your dataset did not have enough dimensions.

• But even in Elements I got the impression that LASSO beats PCR most of the time anyway, and that the main advantage of PCR is when $p > n$ – shadowtalker Dec 11 '14 at 11:25