How to identify the most impactful features in a ML model, i.e. the predictor variables that can drive the biggest change in the target variable?

Question

I have built a machine learning model using Random Forest in Sklearn (RandomForestRegressor). The model has up to 473 predictor variables and 1 target variable (all predictor and target variables are numeric). I wanted to identify the most impactful features, i.e. which predictor variables can result in the biggest change in the target variable.

Using feature importance, I can find the variables that have the most predicting power (53 features account for 90%+ feature importance). However, this is different from being the most impactful. Ideally, I want to be able to make a statement saying "if we are to change all variables by the same percentage, these 3 variables will produce the biggest amount of change in the target variable".

I figured I could use the same concept as in deep learning, i.e. to find the derivatives of all predictor variables and then the variables with the biggest derivative values would be the most impactful variables.

Here are my questions:

• How would I be able to do this when what I have is a sklearn ML model and not an actual math function? Maybe I need to use tensorflow gradienttape?
• What if the predictor variables are not independent? i.e. when one variable changes, others may also change. Can I still make the above statement if I use the derivative method mentioned?

Answer 1

For this purpose I would use the do called partial dependency plots, see the link below to scikit-learn documentation.

Partial dependency plots in scikit-learn

Answer 2

What you want is actually feature importance. The method scikit-learn uses for evaluating it is pretty basic: it sums up the training score gain of all the splits made in all trees using each variable, and then scales the results so that they sum to 1.

The "score gain" of a split is how much the training MSE was reduced by the split, in statistics, that is called $SS_{between}$. In simple words, it is a sum over all observations in that knot, of the squared distance between the prediction considering that split, and the prediction without considering it. It is, to put it simply, the amount impact of that split on predictions. And its sum over all splits that employ one variable, averaged over all trees, is the amount of impact that variable has on predictions.

I understand that maybe a sum of squared distances is not what you were looking for, but you may choose it for its link to the score function, if not because it's so easily served up to you!