# Random forest positive/negative feature importance

I have built a random forest regression model in sklearn. I can obtain a lists of features along with their importances. However, is there a way to determine whether these features have a positive or negative impact on the predicted variable?

• – Ami Tavory Jul 4 '17 at 12:47
• Because the response can be (almost arbitrarily) nonlinear, it doesn't really make sense to me to think of a partial effect as being simply positive or negative. E.g. Is a sawtooth pattern positive or negative? Or a U-shaped curve? – mkt Jul 5 '17 at 9:01
• You need partial dependency plots. A quick google search will turn up how to make them in sklearn. – Matthew Drury Jul 5 '17 at 15:12

## 1 Answer

I wrote a function (hack) that does something similar for classification (it could be amended for regression). The essence is that you can just sort features by importance and then consult the actual data to see what the positive and negative effects are, with the reservation that decision trees are nonlinear classifiers and therefore it's difficult to make statements about isolated feature effects.

If you're truly interested in the positive and negative effects of predictors, you might consider boosting (eg, GradientBoostingRegressor), which supposedly works well with stumps (max_depth=1). With stumps, you've got an additive model.

However, for random forest, you can get a general idea (the most important features are to the left): from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import sklearn.datasets
import pandas
import numpy as np
import pdb
from matplotlib import pyplot as plt
%matplotlib inline

data = sklearn.datasets.load_breast_cancer()
X, y = data.data, data.target
X = pandas.DataFrame(X, columns=data.feature_names)
X_train, X_test, y_train, y_test = train_test_split(X, y)

forest = RandomForestClassifier().fit(X_train, y_train)
forest_prob = forest.predict_proba(X_test)[:,1]
importances = pandas.DataFrame(forest.feature_importances_, index=data.feature_names, columns=['importance'])

def forest_insight(X_val, y_val, thr, forest_prob, importances, nfeat):

dec = map(lambda x: (x> thr)*1,forest_prob)
val_c = X_val.copy()

#scale features for visualization
val_c = pandas.DataFrame(StandardScaler().fit_transform(val_c), columns=X_val.columns)

val_c = val_c[importances.sort('importance', ascending=False).index[0:nfeat]]
val_c['t']=y_val
val_c['p']=dec
val_c['err']=np.NAN

val_c.loc[(val_c['t']==0)&(val_c['p']==1),'err'] = 3#'fp'
val_c.loc[(val_c['t']==0)&(val_c['p']==0),'err'] = 2#'tn'
val_c.loc[(val_c['t']==1)&(val_c['p']==1),'err'] = 1#'tp'
val_c.loc[(val_c['t']==1)&(val_c['p']==0),'err'] = 4#'fn'

n_fp = len(val_c.loc[(val_c['t']==0)&(val_c['p']==1),'err'])
n_tn = len(val_c.loc[(val_c['t']==0)&(val_c['p']==0),'err'])
n_tp = len(val_c.loc[(val_c['t']==1)&(val_c['p']==1),'err'])
n_fn = len(val_c.loc[(val_c['t']==1)&(val_c['p']==0),'err'])

fp = np.round(val_c[(val_c['t']==0)&(val_c['p']==1)].mean(),2)
tn = np.round(val_c[(val_c['t']==0)&(val_c['p']==0)].mean(),2)
tp =  np.round(val_c[(val_c['t']==1)&(val_c['p']==1)].mean(),2)
fn =  np.round(val_c[(val_c['t']==1)&(val_c['p']==0)].mean(),2)

c = pandas.concat([tp,fp,tn,fn],names=['tp','fp','tn','fn'],axis=1)
pandas.set_option('display.max_colwidth',900)
c = c[0:-3]

c.columns = ['TP','FP','TN','FN']
return c

• Could you clarify what your Y axis is? – mkt Jul 5 '17 at 10:55
• It's the mean value of that feature for the cases that are, for example, true positives. – user0 Jul 5 '17 at 14:45
• (After standard scaling.) – user0 Jul 5 '17 at 14:51
• Interesting approach. I think there are areas where it could be misleading (particularly nonlinear relationships where the distribution is highly skewed), but overall it sounds like it could be useful. – mkt Jul 5 '17 at 14:55
• It's crude, and depends on the scaling, but it does quickly give a sense of whether each important variable has a negative or positive effect. – user0 Jul 9 '17 at 16:39