What ways are there to assess variable (feature, covariate) importance in regression models, except for using random forests?

(For instance, using OLS regression, Bayesian parametric regression, etc.?)

  • 2
    $\begingroup$ There's no one way to answer this question. Features are important in the context of a given question or application, not in and of themselves. The naming of the random forest statistic as feature importance is, in my view, a great tragedy. $\endgroup$ – Matthew Drury Jul 22 '16 at 17:57
  • $\begingroup$ @MatthewDrury Yes, but if I use a regression model, how do I quantify how influential individual features are contributing to the regression model? $\endgroup$ – ShanZhengYang Jul 22 '16 at 19:28
  • $\begingroup$ First you should ask yourself why you want to do that, and how you intend to use the information. $\endgroup$ – Matthew Drury Jul 22 '16 at 19:31
  • 2
    $\begingroup$ The "bible" for feature importance are probably Ulrike Gromping's papers. She has an R module called RELAIMPO which implements her computationally intensive method. prof.beuth-hochschule.de/fileadmin/user/groemping/downloads/… One valid, "quick and dirty" heuristic is to run RF regressions and average the t-statistics for each parameter. $\endgroup$ – Mike Hunter Jul 22 '16 at 19:46
  • 1
    $\begingroup$ You can assess the feature importance like in random forest only with other methods. You train your model on several subsets (cross-validation) of the whole data, change the data like in the VIMP method with permutation of the values in one variable and assess performance before and after permutation in the respective holdout data. The difference of the performance is your VIMP. $\endgroup$ – PhilippPro Jul 29 '16 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.