Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 6 down vote favorite
4

I have data with continuous class and I'm searching for good methods to reduce number of attributes. Now I'm using correlation based filters, random forests and Gram–Schmidt algorithm.

What I want to achieve is answer which attributes are more important/relevant to class attribute than others.

By using methods that I mentioned before I can reach this goal, but is there any other good algorithms worth noticing?

share|improve this question
It is impossible to answer this question without knowing what your end goal is. – hadley Jul 11 '10 at 5:48
I altered my question to explain what I want to achieve. – pixel Jul 11 '10 at 11:33
Important or relevant to what?? – hadley Jul 11 '10 at 11:52
Relevant to class attribute... – pixel Jul 11 '10 at 12:25

migrated from stackoverflow.com Sep 26 '11 at 6:23

6 Answers

up vote 4 down vote accepted

My heart will be always with RF, but still you may take a look at Rough Sets. Especially LERS works quite good in case of massively disturbed data.
You may also try with importance obtained from other classifiers, like SVMs or Random Naive Bayes.

share|improve this answer
Actually my data is rather small (~100 examples) and classifier is a result of using PCA (that's why it's continuous). Again can I use LERS, SVM or RNB with continuous classifier? – pixel Jul 10 '10 at 20:37
1  
@pixel For SVM certainly (it is called SVM regression), for RNB there might not be an implementation, but it can be done as in RF. If not, and with LERS you may just partition the decision into a discrete groups -- it is a common trick and will be sufficient for feature selection. – mbq♦ Jul 10 '10 at 21:43
up vote 1 down vote

The Task view on Machine Learning and Statistical Learning is a good starting point for question like this.

share|improve this answer
up vote 1 down vote

Regularised regression with an L1 penatly term has worked well for me (c.f. LASSO and LARS).

share|improve this answer
up vote 1 down vote

I have had good results with ensemble feature selection procedures. For implementation You can take a look at Java-ML library: http://java-ml.sourceforge.net/

For references see for example here: http://www.springerlink.com/content/b00625100k51w7kn/

I believe that this procedires are also readily availiable in R.

share|improve this answer
up vote 1 down vote

I'm a big fan of the rfe function in the caret package. You can easily use it to cross-validate feature importance ratings from a random forest, a linear model, a bagged-tree model, a naive bayesian model, or any other algorithm that returns a measure of variable importance.

You can read more here.

share|improve this answer
up vote 0 down vote

Principal Component Analysis is a fairly common technique used to reduce the dimension of sampled data. You can find a very good implementation in R.

share|improve this answer
Of course PCA reduces number of dimensions but also looses information about original attributes - it doesn't provide answer which attributes are 'better' than others. – pixel Jul 10 '10 at 19:44
1  
In PCA you can look at the composition of components, so you can see what attributes were mainly used for creating first PCs. Nevertheless it is useless for data with complex or subtle dependencies. – mbq♦ Jul 10 '10 at 21:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.