As a rough draft of a solution, I suppose it would be useful to shuffle one at a time, each of the columns of predictor variables, rerun predictions, then check the effect on the metric like MSE versus unshuffled. You could do it a few times to get a mean effect (mean of MSE) and a variance and a confidence interval for each predictor. Overlapping confidence intervals would indicate some uncertainty as to true variable importance.
It seems like we should borrow some aspects of the display of the R linear regression's coefficient table, which shows for each coefficient the mean, std error, t-value, probability, and graphical "star rating" of the likely significance (3=highest, 0=lowest).
I recognize that when a model has 2nd order terms and higher, there are potentially more terms in a linear model than there are predictors in the dataset. So variable importance is not really measuring the same thing as the R coefficient table (but there are some similarities). The purpose of this question is to compute VI for neural networks, mostly, but why not every model design too.
Has this concept been worked out more fully and more formally before?
Similar but different question: How are the standard errors of coefficients calculated in a regression? How are the standard errors of coefficients calculated in a regression?
The Fast.AI library for PyTorch had a function var_importance() which seemed to do something like this.
I want to understand the theoretical basis for model-architecture-independent variable importance computation better before reading code.
