Getting feature importance of a black box model

Question

I have a black box random forest model where I can only feed some input data in matrix form and receive the predictions in a vector. Is it possible to obtain any information about the feature importances by just feeding it data and looking at how the predictions change? I also have access to the data the model was trained on, but have no access to any other information of the model itself.

I've previously done this with a black box linear regression model by feeding it identity matrices, but does that not work in the case of a random forest?

Answer 1

A model agnostic way to assess variable importance is permutation importance. Model agnostic means it works for any type of model.

How is it calculated?

1) Choose evaluation metric and calculate original performance on your query data, e.g. an independent validation set.

2) Select input x. Shuffle column x in the data and (without refitting) get predictions. Calculate performance on shuffled data. Variable importance of x is assessed by calculating difference (or relative change) in performance before and after shuffling.

3) Repeat 2) for all covariables of interest.

You can use R packages DALEX, iml or flashlight (mine) to calculate such measure.