2
$\begingroup$

My goal is to get the feature importance for multiple regression. I have a data set with some multicollinearity. I found two methods to solve this problem.

The first one is the Shapley value. Shapley value computes the regression using all possible combinations of predictors and computes the R$^2$ for each model. Then for each predictor, the average improvement will be calculated that is created when adding that variable to a model.

As an example, I use the Boston data set (I found this result on different data sets).

library(relaimpo)
library(MASS)
data(Boston)

mod_lm = lm(medv ~ ., data = Boston)

calc.relimp(mod_lm, type = c("lmg"))

The relative importance is scaled to 100%. In the case, Islat is ~5.5 times as important as tax (I just show the relative importance but not the whole output).

    Relative importance metrics: 

               lmg
crim    0.03669197
zn      0.03349910
indus   0.05101231
chas    0.02139033
nox     0.04511571
rm      0.25259867
age     0.02975151
dis     0.04087946
rad     0.03190385
tax     0.04967225
ptratio 0.10602348
black   0.03093651
lstat   0.27052485

Ridge regression adds a penalty to all beta coefficients. The size of the penalty is dependent on the RMSE in the cross-validation or bootstrap. I standardize the data to get the relative importance.

library(caret) 

Boston_scaled = scale(Boston)

fit = train(medv~., data = Boston_scaled, 
            method = "glmnet", 
            preProcess = NULL, 
            tuneGrid = expand.grid(alpha = 0, lambda = seq(0,1, by = 0.1)))
                   

coef(fit$finalModel, fit$bestTune$lambda)

In this case, Islat is only 3.5 times as important as tax.

14 x 1 sparse Matrix of class "dgCMatrix"
                                    1
(Intercept) -0.0000000000000006334209
crim        -0.0819019359606095592730
zn           0.0828741063806194971919
indus       -0.0283478772157603683968
chas         0.0800825186065121513712
nox         -0.1501007822559678184238
rm           0.3064458129978267497684
age         -0.0114206187124149035478
dis         -0.2561703730728009387185
rad          0.1455422731561680227408
tax         -0.1053881725850110001597
ptratio     -0.2012584716226800130023
black        0.0900702979932691133458
lstat       -0.3668119207994711694631

Since both metrics are standardized I should get similar results. Can someone explain the difference in the results?

$\endgroup$
11
  • 3
    $\begingroup$ You're comparing ridge regression coefficients to shapley values from a random forest? These measure quite different things, so there's no reason they should tell the same story. $\endgroup$ Commented Aug 7, 2019 at 14:17
  • $\begingroup$ No, Shapely values were calculated based on the OLS (see mod_lm) and the standardized Betacoeficients based on ridge regression. Both are the same methods besides the shrinking parameter in the ridge regression. $\endgroup$
    – Banjo
    Commented Aug 7, 2019 at 14:40
  • $\begingroup$ It is Shapley, and not Shapely. Edited accordingly. $\endgroup$
    – Nick Cox
    Commented Aug 7, 2019 at 14:52
  • 3
    $\begingroup$ "Which of both methods is wrong". There's no fundamental concept of "correct" here. No one agrees on what something we call "feature importance" should measure, and people barely agree on what properties such a measure should have. It's a coincidence that both of these methodologies are labeled a "feature importance", they simply measure different things. $\endgroup$ Commented Aug 7, 2019 at 15:47
  • 1
    $\begingroup$ @Banjo From that perspective, they just measure something different. I would like to note though, that the definition that you offer of "feature importance", just passes the responsibility of precision to defining what you mean by "drive the target variable", which is also an undefined term / phrase. That's just not rigorous enough to define a real concept in statistics or machine learning. $\endgroup$ Commented Aug 13, 2019 at 4:43

1 Answer 1

2
$\begingroup$

There are several reasons why the two sets of values you computed are not the same:

$R_2$ contributions are not partial correlation or regression coefficients

The package you use to compute Shapley values computes contribution to the $R^2$. This is a value between 0 and 1. Ridge regression coefficients on standardized data are partial correlations and can take any value between -1 and 1. Accounting for this takes away a lot (but surely not all) of the discrepancies:

ridge_coefs <- coef(fit$finalModel, fit$bestTune$lambda)
shapley_vals <- calc.relimp(mod_lm, type = c("lmg"))$lmg
cor(ridge_coefs[-1], shapley_vals, method = "spearman")
[1] -0.4505495
cor(abs(ridge_coefs[-1]), shapley_vals, method = "spearman")
[1] 0.6703297

Perhaps you also want to take the square root of the Shapley, but for the Spearman correlation this will not matter.

Computation of Shapley values differs from that of partial correlations

Shapley values are not partial correlations. The Shapley value gives a weighted average of the increases in $R^2$ when a given predictor is added to all possible sets of predictors excluding that predictor (as computed by the package you use, note there are many ways to compute Shapley values, see e.g., Chen et al., 2022: Algorithms to estimate Shapley value feature attributions). This procedure is different than the (penalized) partial regression coefficients (partial correlations in your case) computed in a ridge regression, which can be computed as follows once the value of $\lambda$ has been chosen:

$$\hat{\beta} = (X^\top X + \lambda I)^{-1}X^\top y$$

Regularization

The ridge coefficients are shrunken towards 0, the Shapley values you computed are not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.