I have built a random forest regression model in sklearn. I can obtain a lists of features along with their importances. However, is there a way to determine whether these features have a positive or negative impact on the predicted variable?

  • $\begingroup$ See this question on SO. $\endgroup$
    – Ami Tavory
    Jul 4, 2017 at 12:47
  • $\begingroup$ Because the response can be (almost arbitrarily) nonlinear, it doesn't really make sense to me to think of a partial effect as being simply positive or negative. E.g. Is a sawtooth pattern positive or negative? Or a U-shaped curve? $\endgroup$
    – mkt
    Jul 5, 2017 at 9:01
  • 1
    $\begingroup$ You need partial dependency plots. A quick google search will turn up how to make them in sklearn. $\endgroup$ Jul 5, 2017 at 15:12

1 Answer 1


I wrote a function (hack) that does something similar for classification (it could be amended for regression). The essence is that you can just sort features by importance and then consult the actual data to see what the positive and negative effects are, with the reservation that decision trees are nonlinear classifiers and therefore it's difficult to make statements about isolated feature effects.

If you're truly interested in the positive and negative effects of predictors, you might consider boosting (eg, GradientBoostingRegressor), which supposedly works well with stumps (max_depth=1). With stumps, you've got an additive model.

However, for random forest, you can get a general idea (the most important features are to the left):

enter image description here

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import sklearn.datasets
import pandas
import numpy as np
import pdb
from matplotlib import pyplot as plt
%matplotlib inline

data = sklearn.datasets.load_breast_cancer()
X, y = data.data, data.target
X = pandas.DataFrame(X, columns=data.feature_names)
X_train, X_test, y_train, y_test = train_test_split(X, y)

forest = RandomForestClassifier().fit(X_train, y_train)
forest_prob = forest.predict_proba(X_test)[:,1]
importances = pandas.DataFrame(forest.feature_importances_, index=data.feature_names, columns=['importance'])

def forest_insight(X_val, y_val, thr, forest_prob, importances, nfeat):

    dec = map(lambda x: (x> thr)*1,forest_prob)
    val_c = X_val.copy()

    #scale features for visualization
    val_c = pandas.DataFrame(StandardScaler().fit_transform(val_c), columns=X_val.columns)

    val_c = val_c[importances.sort('importance', ascending=False).index[0:nfeat]]

    val_c.loc[(val_c['t']==0)&(val_c['p']==1),'err'] = 3#'fp'
    val_c.loc[(val_c['t']==0)&(val_c['p']==0),'err'] = 2#'tn'
    val_c.loc[(val_c['t']==1)&(val_c['p']==1),'err'] = 1#'tp'
    val_c.loc[(val_c['t']==1)&(val_c['p']==0),'err'] = 4#'fn'

    n_fp = len(val_c.loc[(val_c['t']==0)&(val_c['p']==1),'err'])
    n_tn = len(val_c.loc[(val_c['t']==0)&(val_c['p']==0),'err'])
    n_tp = len(val_c.loc[(val_c['t']==1)&(val_c['p']==1),'err'])
    n_fn = len(val_c.loc[(val_c['t']==1)&(val_c['p']==0),'err'])

    fp = np.round(val_c[(val_c['t']==0)&(val_c['p']==1)].mean(),2)
    tn = np.round(val_c[(val_c['t']==0)&(val_c['p']==0)].mean(),2)
    tp =  np.round(val_c[(val_c['t']==1)&(val_c['p']==1)].mean(),2)
    fn =  np.round(val_c[(val_c['t']==1)&(val_c['p']==0)].mean(),2)

    c = pandas.concat([tp,fp,tn,fn],names=['tp','fp','tn','fn'],axis=1)
    c = c[0:-3]

    c.columns = ['TP','FP','TN','FN']
    return c
  • $\begingroup$ Could you clarify what your Y axis is? $\endgroup$
    – mkt
    Jul 5, 2017 at 10:55
  • $\begingroup$ It's the mean value of that feature for the cases that are, for example, true positives. $\endgroup$ Jul 5, 2017 at 14:45
  • $\begingroup$ (After standard scaling.) $\endgroup$ Jul 5, 2017 at 14:51
  • $\begingroup$ Interesting approach. I think there are areas where it could be misleading (particularly nonlinear relationships where the distribution is highly skewed), but overall it sounds like it could be useful. $\endgroup$
    – mkt
    Jul 5, 2017 at 14:55
  • $\begingroup$ It's crude, and depends on the scaling, but it does quickly give a sense of whether each important variable has a negative or positive effect. $\endgroup$ Jul 9, 2017 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.