Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Yesterday I asked this question in which I had 180 subjects with 500 features each. While I was sure that dimensional-reduction is a must in this case (500 features), most of the answers I got said that 500 are not too many.
So, My question is: Is there any rule of thumb when one should use dimensional-reduction before the classifier? How many features is too many? (I guess it is depends upon the ratio between the number of subjects and features. Isn't it?)
Rather than asking "when use" let's look at "why use" - I believe this nicely leads us to the "when" answer.
My understanding is that dimensionality reduction is mainly done to
"Unacceptable" is, obviously, defined solely by the task at hand. Modern computers can handle a lot of computations and store a lot of data - which is why, I think, you was told that 500 features is not too much. There are few other reasons for dimensionality reduction I can think of:
matrix inversion problems - an algorithm can build a matrix from sample set, and if some features are interdependent this makes the marix non-invertible. But in practice it's not a big deal and gets circumvented via Moore-Penrose pseudoinverse so, in my view, this one should not be the reason for dimensionality reduction.
data visualization - the rule of thumb here is to extract features until you're left with a maximum of two, due to a deficiency in human cognition :)
As far as I know, we don't have a rule of thumb regarding when to use dimensional reduction. I'm also thinking that, is depends upon the ratio between the number of subjects and features. Also other factors such as processing power of the system you are going to deploy your learning algorithm, might have to consider.
Further, dimensional reduction techniques such as sparse auto-encoder are capable of finding interesting patterns in the data, hence improve the accuracy of algorithms. Therefore one might think that it is always better to use a dimensional reduction method.
The number of features is not the only reason for reduction. It is also important to check what are these features.
Although this is a computer science oriented site, the issues of memory and run time are relevant but they shouldn't be the only focus of many of the learning tasks.
When you are selecting your features, you should have some kind of hypothesis regarding what is relevant for the task in hand. If you selected you features in a random way, or in a way that is not related to the task you wish to learn, it is OK to continue using "random" methods to reduce this number. But if you had some hypothesis about the features, I would try to keep as many of them as possible in the learning process.
In general, the better understanding you have and the better planning of your task regarding what are the best features to learn with, the better your results will be.
If the complexity of your model or classifier trained on those n features scales badly (e.g. the number of parameters grows as O(n^3)), then even 500 features can be a problem. Not only because the optimization takes longer, but also because you might not have enough data to constrain your parameters, which would lead to overfitting.
By reducing model complexity, dimensionality reduction can therefore also act as a means of regularization.
I saw another very interesting usage case for dimensionality reduction in a video from stanford a while ago. They scanned a bunch of people with a body scanner, and used that to generate 3d models. After they had a bunch of data they applied dimensionality reduction to reduce the amount of variables they had to work with. And modifying those variables allowed them to quickly change the height/weight/gender of the resulting 3d models.
