12
$\begingroup$

What are the applications or advantages of dimension reduction regression (DRR) or supervised dimensionality reduction (SDR) techniques over traditional regression techniques (without any dimensionality reduction)? These class of techniques finds a low-dimensional representation of the feature set for the regression problem. Examples of such techniques include Sliced Inverse Regression, Principal Hessian Directions, Sliced Average Variance Estimation, Kernel Sliced Inverse Regression, Principal Components Regression, etc.

  1. In terms of cross-validated RMSE, if an algorithm performed better on a regression task without any dimensionality reduction, then what is the real use of the dimensionality reduction for regression? I don't get the point of these techniques.

  2. Are these techniques by any chance used to reduce space and time complexity for regression? If that is the primary advantage, some resources on complexity reduction for high-dimensional datasets when this techniques are put to use would be helpful. I debate this with the fact that running a DRR or SDR technique itself requires some time and space. Is this SDR/DRR + Regression on a low-dim dataset faster than only regression on a high-dim dataset?

  3. Has this setting been studied out of abstract interest only, and does not have a good practical application?

As a side thought: at times there are assumptions that the joint distribution of the features $X$ and the response $Y$ lies on a manifold. It makes sense to learn the manifold from the observed sample in this context for solving a regression problem.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
5
$\begingroup$

According to the manifold hypothesis, the data is assumed to lie on a low-dimensional manifold, the implication being that the residual is noise, so if you do your dimensionality reduction correctly, you should improve performance by modeling the signal rather than the noise. It's not just a question of space and complexity.

Improve this answer
$\endgroup$
5
  • $\begingroup$ but I do not see techniques like SIR doing better after dimensionality reduction on a robust basis. Correct me if am wrong or if you know of a SDR/DDR technique that can find this signal better-in a regression setting, let me know of what technique (name) it is. $\endgroup$
    – hearse
    Dec 10, 2014 at 23:46
  • $\begingroup$ Of course it depends on the regression algorithm, and the intrinsic dimensionality of the data. I can't speak for SIR in particular, but here's a paper that compares various regression algorithms on the MNIST dataset, which is low dimensional. Maybe you could share some troublesome data so people can take a crack at it. $\endgroup$
    – Emre
    Dec 11, 2014 at 0:13
  • $\begingroup$ What is "the manifold hypothesis"? $\endgroup$
    – amoeba
    Dec 11, 2014 at 9:24
  • 1
    $\begingroup$ The hypothesis that high dimensional data tends to lie in the vicinity of a low dimensional manifold. $\endgroup$
    – Emre
    Dec 11, 2014 at 9:27
  • $\begingroup$ I wonder if this stuff is similar to neural networks and nonlinear multidimensional scaling in that it "sounds like" it should be great everywhere but in practice does well in a more limited set of cases $\endgroup$
    – shadowtalker
    Dec 11, 2014 at 11:28
6
$\begingroup$

The purpose of dimensionality reduction in regression is regularization.

Most of the techniques that you listed are not very well known; I have not heard about any of them apart from principal components regression (PCR). So I will reply about PCR but expect that the same applies to the other techniques as well.

The two key words here are overfitting and regularization. For a long treatment and discussion I refer you to The Elements of Statistical Learning, but very briefly, what happens if you have a lot of predictors ($p$) and not enough samples ($n$) is that standard regression will overfit the data and you will construct a model that seems to have good performance on the training set but actually has very poor performance on any test set.

In an extreme example, when number of predictors exceeds number of samples (people refer to it as to $p>n$ problem), you can actually perfectly fit any response variable $y$, achieving seemingly $100\%$ performance. This is clearly nonsense.

To deal with overfitting one has to use regularization, and there are plenty of different regularization strategies. In some approaches one tries to drastically reduce the number of predictors, reducing the problem to the $p\ll n$ situation, and then to use standard regression. This is exactly what principal components regression does. Please see The Elements, sections 3.4--3.6. PCR is usually suboptimal and in most cases some other regularization methods will perform better, but it is easy to understand and interpret.

Note that PCR is not arbitrary either (e.g. randomly keeping $p$ dimensions is likely to perform much worse). The reason for this is that PCR is closely connected to ridge regression, which is a standard shrinkage regularizer that is known to work well in a large variety of cases. See my answer here for the comparison: Relationship between ridge regression and PCA regression.

To see a performance increase compared to standard regression, you need a dataset with a lot of predictors and not so many samples, and you definitely need to use cross-validation or an independent test set. If you did not see any performance increase, then perhaps your dataset did not have enough dimensions.

Related threads with good answers:

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Given his publications it's safe to assume he knows this. $\endgroup$
    – Emre
    Dec 11, 2014 at 9:28
  • $\begingroup$ Thanks, @Emre, I had no idea who the OP was. I might have misunderstood the question, but after re-reading it now I don't see how I can interpret it differently. If one is asking what is the practical advantage of PCR, then the answer is regularization; PCR is actually closely related to ridge regression, which is one of the most standard regularization methods. $\endgroup$
    – amoeba
    Dec 11, 2014 at 9:38
  • $\begingroup$ But even in Elements I got the impression that LASSO beats PCR most of the time anyway, and that the main advantage of PCR is when $p > n$ $\endgroup$
    – shadowtalker
    Dec 11, 2014 at 11:25
  • $\begingroup$ @ssdecontrol: I agree. I think the consensus is that PCR is pretty much non-competitive and almost always there are better approaches. This is also what I wrote in my answer (didn't I?), but the question was specifically about dimensionality reduction of predictors and about what its purpose might be. My answer is that the purpose is regularization. $\endgroup$
    – amoeba
    Dec 11, 2014 at 11:31
  • $\begingroup$ Understood. But I think we can agree that the question is loaded specifically to challenge its usefulness given that it isn't actually the best way to regularize despite its intuitive appeal $\endgroup$
    – shadowtalker
    Dec 11, 2014 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.