1
$\begingroup$

I trained decision tree both in python and R, but I think the way feature importance is calculated in R may be wrong. Following is the sample code which you can use to reproduce the problem. Let's say I am predicting Income of population of 1000 based on gender and country.

x = data.frame(gender=sample(c("M","F"),n,T), country=sample(c("A","I"),n,T))
x$income = ifelse(x$gender=="M", rnorm(n, 100, 10), rnorm(n, 80, 10))
x$income = x$income + ifelse(x$country=="A", rnorm(n, 100, 10), rnorm(n, 80, 10))
write.csv(x, "data.csv")

Then lets fit a decision tree in R with max depth of 1.

fit = rpart(income~., data = x, control=rpart.control(maxdepth=1))
caret::varImp(fit)
fit

I get the following feature importance

country 0.2507630, and
gender 0.2424981

For the tree split only at country

1) root 1000 407373.4 180.5759  
  2) country=I 481 147999.6 170.0772 *  
  3) country=A 519 157219.6 190.3060 *  

When I try again with Max depth of 2, I get feature importance as

country 0.2507630, and
gender 0.8874599

For the tree split first at country as before and then at gender

1) root 1000 407373.40 180.5759    
  2) country=I 481 147999.60 170.0772    
     4) gender=F 232  40082.49 159.2805 *  
     5) gender=M 249  55676.09 180.1367 *  
  3) country=A 519 157219.60 190.3060    
     6) gender=F 248  57546.77 180.4749 *  
     7) gender=M 271  53767.73 199.3028 *  

However, if I run similar code in python

from io import StringIO
from sklearn.tree import DecisionTreeRegressor
from sklearn.tree.export import export_graphviz
from IPython.display import Image 
from sklearn import tree
import pydot
import pandas as pd

data = pd.read_csv("data.csv")

dtree=DecisionTreeRegressor(max_depth= 1)
X = data[["gender", "country" ]]
X["gender"] = X["gender"] == 'M'
X["country"] = X["country"] == 'A'
y = data[['income']]
dtree.fit(X,y)

# Export as dot file
export_graphviz(dtree, out_file='tree.dot', 
                feature_names = X.columns,filled=True, rounded=True,
                special_characters=True)

(graph,) = pydot.graph_from_dot_file('tree.dot')
graph.write_png('tree.png')
# Display in jupyter notebook
from IPython.display import Image
Image(filename = 'tree.png')

for max depth of 1, I get feature importance as

gender  0, and  
country 1 

and for max depth of two

gender  0.49, and  
country 0.51  

Now I have following two questions
1) In R, when I select max depth = 1 and split happened only at country column, then why it still gives feature importance value for gender. Even though gender is not even part of the final model. For e.g. in python it gives gender as 0 var importance.
2) Secondly, why in R, the feature importance of gender column became greater than that of country column? As the country column was more important because the initial split happened at country and not at gender. Similar to the values we got for python.
One of my colleague pointed out that in R, the feature importance of each column is calculated at each split. For e.g. the some feature importance value of gender and country will be calculated at first split. Then again this happens at second split. But since we already had a split based on country there will not be any information gain based on country but would be there for gender. And in the end all these importance are summed. Hence we get more value for the ones that were used at a later stage for split.

$\endgroup$

closed as off-topic by Robert Long, mdewey, Siong Thye Goh, Peter Flom Apr 11 at 13:12

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Robert Long, mdewey, Siong Thye Goh, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ I found this link, which somewhat answers my first question. But it still does not answers the second question. $\endgroup$ – MNA Apr 8 at 7:21