I have a black box random forest model where I can only feed some input data in matrix form and receive the predictions in a vector. Is it possible to obtain any information about the feature importances by just feeding it data and looking at how the predictions change? I also have access to the data the model was trained on, but have no access to any other information of the model itself.
I've previously done this with a black box linear regression model by feeding it identity matrices, but does that not work in the case of a random forest?