I have a black box random forest model where I can only feed some input data in matrix form and receive the predictions in a vector. Is it possible to obtain any information about the feature importances by just feeding it data and looking at how the predictions change? I also have access to the data the model was trained on, but have no access to any other information of the model itself.

I've previously done this with a black box linear regression model by feeding it identity matrices, but does that not work in the case of a random forest?

  • 1
    $\begingroup$ That won't work for random forests or for many other models. I don't think it would work for any model that accounts for interactions between the input features because, in such models, you need to change two variables at a time to see the effect. In addition, with something like a random forest, there are threshold effects to account for. $\endgroup$ Commented Oct 5, 2019 at 22:43

1 Answer 1


A model agnostic way to assess variable importance is permutation importance. Model agnostic means it works for any type of model.

How is it calculated?

1) Choose evaluation metric and calculate original performance on your query data, e.g. an independent validation set.

2) Select input x. Shuffle column x in the data and (without refitting) get predictions. Calculate performance on shuffled data. Variable importance of x is assessed by calculating difference (or relative change) in performance before and after shuffling.

3) Repeat 2) for all covariables of interest.

You can use R packages DALEX, iml or flashlight (mine) to calculate such measure.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.