2
$\begingroup$

I'm training decission trees for a project in which I want to predict the behavior of one variable according to the others (there are about 20 other variables).

Some time ago I was using simple logistic regression models in another project (using R). There is a nice feature in R where you can see the statistical significance of every variable introduced in the model. This helps in simplifying the model by removing not meaningful variables.

I wonder if there is a way to do the same with Decission trees (this time I'm using Python and scikit-learn). Can we see which variables are really important for a trained model in a simple way?

Thanks!

$\endgroup$
  • 3
    $\begingroup$ Its not related to your main question, but it is not advisable to remove variables from regression models based on p-values of significance scores. That is not what those measures are for. $\endgroup$ – Matthew Drury Sep 5 '17 at 14:48
  • 1
    $\begingroup$ The nice thing about decision trees is that they find out by themselves which variables are important and which aren't. You couldn't build a tree if the algorithm couldn't find out which variables are important to predict the outcome, you wouldn't know what to branch on. Now the mathematical principles behind that selection are different from logistic regressions and their interpretation of odds ratios. $\endgroup$ – David Ernst Sep 5 '17 at 15:07
1
$\begingroup$

You can use the following method to get the feature importance. First of all built your classifier.

clf= DecisionTreeClassifier()

now

clf.feature_importances_

will give you the desired results.

The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance.

$\endgroup$
  • $\begingroup$ Yes great!!! actually it does! I would love to know how those factors are actually computed. $\endgroup$ – Ambesh Sep 6 '17 at 12:25
2
$\begingroup$

The concept of statistical significance doesn't exist for decisions trees. The closest tool you have at your disposal is called "Gini impurity" which tells you whether a variable is more or less important when constructing the (bootstrapped) decision tree. In R, a ready to use method for it is called varImpPlot in the package randomForest - not sure about Python. But I hope at least that helps you in terms of what to google.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.