I trained decision tree both in python and R, but I think the way feature importance is calculated in R may be wrong. Following is the sample code which you can use to reproduce the problem. Let's say I am predicting Income of population of 1000 based on gender and country.
x = data.frame(gender=sample(c("M","F"),n,T), country=sample(c("A","I"),n,T)) x$income = ifelse(x$gender=="M", rnorm(n, 100, 10), rnorm(n, 80, 10)) x$income = x$income + ifelse(x$country=="A", rnorm(n, 100, 10), rnorm(n, 80, 10)) write.csv(x, "data.csv")
Then lets fit a decision tree in R with max depth of 1.
fit = rpart(income~., data = x, control=rpart.control(maxdepth=1)) caret::varImp(fit) fit
I get the following feature importance
country 0.2507630, and gender 0.2424981
For the tree split only at country
1) root 1000 407373.4 180.5759 2) country=I 481 147999.6 170.0772 * 3) country=A 519 157219.6 190.3060 *
When I try again with Max depth of 2, I get feature importance as
country 0.2507630, and gender 0.8874599
For the tree split first at country as before and then at gender
1) root 1000 407373.40 180.5759 2) country=I 481 147999.60 170.0772 4) gender=F 232 40082.49 159.2805 * 5) gender=M 249 55676.09 180.1367 * 3) country=A 519 157219.60 190.3060 6) gender=F 248 57546.77 180.4749 * 7) gender=M 271 53767.73 199.3028 *
However, if I run similar code in python
from io import StringIO from sklearn.tree import DecisionTreeRegressor from sklearn.tree.export import export_graphviz from IPython.display import Image from sklearn import tree import pydot import pandas as pd data = pd.read_csv("data.csv") dtree=DecisionTreeRegressor(max_depth= 1) X = data[["gender", "country" ]] X["gender"] = X["gender"] == 'M' X["country"] = X["country"] == 'A' y = data[['income']] dtree.fit(X,y) # Export as dot file export_graphviz(dtree, out_file='tree.dot', feature_names = X.columns,filled=True, rounded=True, special_characters=True) (graph,) = pydot.graph_from_dot_file('tree.dot') graph.write_png('tree.png') # Display in jupyter notebook from IPython.display import Image Image(filename = 'tree.png')
for max depth of 1, I get feature importance as
gender 0, and country 1
and for max depth of two
gender 0.49, and country 0.51
Now I have following two questions
1) In R, when I select max depth = 1 and split happened only at country column, then why it still gives feature importance value for gender. Even though gender is not even part of the final model. For e.g. in python it gives gender as 0 var importance.
2) Secondly, why in R, the feature importance of gender column became greater than that of country column? As the country column was more important because the initial split happened at country and not at gender. Similar to the values we got for python.
One of my colleague pointed out that in R, the feature importance of each column is calculated at each split. For e.g. the some feature importance value of gender and country will be calculated at first split. Then again this happens at second split. But since we already had a split based on country there will not be any information gain based on country but would be there for gender. And in the end all these importance are summed. Hence we get more value for the ones that were used at a later stage for split.