How to do feature selection by using Classification and Regression Tree? As I know, splitting data in decision tree can use Gini Index or Entropy, but it can't be used in feature selection. So how I set a threshold to split? Is it up to us to adjust the threshold? I read this journal but I still don't understand
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ Are trying to kill performance? Feature selection will help you in doing that $\endgroup$– utobiNov 16, 2016 at 8:20
-
$\begingroup$ what do you mean? I attach a journal, if you mind to read $\endgroup$– user137192Nov 16, 2016 at 8:34
-
1$\begingroup$ Yes thanks for the proceeding paper. What I mean is that, the only real usefulness of feature selection is when you have a really super high number of features and their screening is infeasible. If you don't have such a high number, don't worry about feature selection, CART, or better random forests (RF) will work any way. $\endgroup$– utobiNov 16, 2016 at 9:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
14
Feature selection is just deciding which variable to include in your model. In case of CART (and most Machine Learning methods) feature selection is done by the model itself.
So how to do feature selection? Just run the algorithm and let the Gini Index or Entropy decide which variable is useful to include in the tree. Here is your feature selection. Make a plot of the tree to find the included variables.
In other models (read non Machine Learning methods) you must do 1) feature selection yourself or 2) decide on an algorithm to do the feature selection for you. An example of 1) is using common sense or a professional opinion to think of features which probably explain the variable you want to explain. Examples of 2) are: stepwize regression based on information criteria or estimate the model using the LASSO method.
ps what utobi meant in his comment (I think): CART does the feature selection for you, so the more variables/features you discard before running the CART algorithm, the higher the chances that you discard an important variable. This leads to a loss in performance. For CART it does not matter if you use 10 or 1000 features, it searches for the best of those 10 or 1000 features (however computation time can be an issue with many features).
-
$\begingroup$ maybe I should edit my question, what I mean is how to use CART for feature classification? $\endgroup$ Nov 16, 2016 at 9:59
-
1$\begingroup$ Never heard of feature classification. If you meant feature selection, just run the algorithm and look at which variables are included in the resulting tree, but this was already in my answer. Please elaborate a bit more: what is your problem, what do you want to do. Otherwise it is very hard to help you. $\endgroup$– Marcel10Nov 16, 2016 at 10:10
-
$\begingroup$ I am sorry to bother you. Yes I have run my algorithm, actually it is a toolbox, because of that I don't much understand. $\endgroup$ Nov 16, 2016 at 12:18
-
$\begingroup$ My question is about the threshold, how I can find it, the threshold to split data I mean. $\endgroup$ Nov 16, 2016 at 12:48
-
$\begingroup$ @tkmn Well, the default metric in CART is the Gini impurity. A node is called pure if it only contains observations of one type. (So if we have two classes, A & B, and a certain node only contains class A observations, this node is pure.) The CART algorithm calculates the Gini impurity for each potential split; then it makes the split which makes the node the purest. So which treshold are you talking about? You can use other measures of impurity, such as entropy. But the results would be (or should be) comparable. $\endgroup$– Marcel10Nov 16, 2016 at 13:12