I am classifying different texts and I wondering about some features that are highly correlated. I have 49 features. Some features are absolute counters (integers) but most features are relative counters(float between 0-1). I am running F-score (univariate) and I am getting the following three features with the highest scores: 1-fourth root of number of word forms, 2- number of word forms and 3-number of sentences. I am running a feature ranking based on extremely randomized trees (scikit-learn ensemble forests) and I am getting exactly the same three features as the highest ranking features. The ranking based on randomized trees is using bootstrapping and GINI. In the F-scores results I can understand that highly correlated features may have the highest ranking because it is univariate based (measure only one feature at the time). In the random tree ranking I was expecting that only one of features related to the "length of the text" should have a high rank and the others should have a lower and the correlation problem should be solved. But the results are not according to my expectations. I must be doing something wrong! Could it be realted to the fact that all three features are integer counters (values in 1000-range) and the other features are relative counters(0-1). But as I undertand ranking based on random trees should be able to handle large discrepancies between the features. My question is how should I handle this issue. Should I discard some features? How can I find out the best feature that characterizes the text length??? Any help here is appreciated!
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
If I understand you correctly, there is no mystery; what could be happening is that the two highly correlated measures are very close to each other in terms of how could the split is.
In your case, though, it isn't very close, it's exact. In particular, total number of words and fourth root of words will split identically, since they are monotonic (given that number of words must be positive). By including both in a tree, you are asking the tree algorithm to pick one at random.
answered Nov 4, 2012 at 18:08
-
$\begingroup$ Thanks Peter ! the forest ensemble will pick the two features randomly if they provide the same split. There should not be any gains in keeping both features. I could remove one of them and I should get the same classification result. $\endgroup$ Nov 4, 2012 at 18:22
-
$\begingroup$ Yes, you should. Try it, and if you do not get the same results, then let us know. That would be odd. $\endgroup$ Nov 4, 2012 at 18:23