Comparison of regression models in terms of the importance of variables

Question

I would like to compare models (multiple regression, LASSO, Ridge, GBM) in terms of the importance of variables. But I'm not sure if the procedure is correct, because the values ​​obtained are not on the same scale.

In multiple regression and GBM values ​​range from 0 - 100 using varImp from the caret package. The calculation of this statistic is distinct in each of the methods.

Linear Models: the absolute value of the t-statistic for each model parameter is used.

Boosted Trees: this method uses the same approach as a single tree, but sums the importance of each boosting iteration.

While for LASSO and Ridge the values ​​are from 0.00 - 0.99, calculated with the function:

varImp <- function (object, lambda = NULL, ...) {
  beta <- predict (object, s = lambda, type = "coef")
  if (is.list (beta)) {
    out <- do.call ("cbind", lapply (beta, function (x)
      x [, 1])))
    out <- as.data.frame (out)
  } else
    out <- data.frame (Overall = beta [, 1])
  out <- abs (out [rownames (out)! = "(Intercept)",, drop = FALSE])
  out
}

Which was obtained here: Caret package - glmnet variable importance

I was guided by other questions on the forum, but could not understand why there is the difference between the scales. How can I make these measurements comparable?

Answer 1

The importance measures here are unlikely to be adapted into a scale that can be compared. Rather, the best way to compare these models is to compare the rank order of the input/predicting variables.

It is also worth noting that the linear/lasso/ridge models are fairly different in what they represent compared to boosted tree ensemble. In particular, the boosted trees are based on series of if-then statements in the underlying decision trees that are necessarily non-linear in their construction. By contrast, linear/lasso/ridge models are based on individual linear coefficients representing each variable (with different constraints on estimation across linear, lasso, and ridge models). Thus, they aren't really comparing the same kinds of models.

A discussion of a related issue is outlined in Groemping (2009).

Again, if a comparison between these is still of interest, the best comparison in this case is likely between the rank ordering of the predicting variables as implied by the differently scaled Caret package outputs.

Grömping, U. (2009). Variable importance assessment in regression: linear regression versus random forest. The American Statistician, 63(4), 308-319.