Different variable importance results with stabsel and mboost

Question

I'm using glmboost in the mboost package to fit a boosted regression using linear models as the base learner. There are 13200 observations and about 75 variables, and I want to get a measure of the importance of each variable.

At the moment, I'm exploring the following two options:

1. The boosted model object returned by glmboost includes information on the selection probabilities of the variables, ie how frequently they are selected by the boosting algorithm.

2. I can use stabsel, from the package of the same name, to identify the important variables. This uses a resampling approach to perturb the data, and the output is the frequency with which each variable appears in the resampled models (if I understand correctly).

The problem is that these two methods are giving radically different results. This is the output from 1:

> summary(php.glmb)

         Generalized Linear Models Fitted via Gradient Boosting

Call:
glmboost.formula(formula = p_hp ~ ., data = hdata.php.trn)

....

Selection frequencies:
degc238   etc48    etc7   per45   bar25   bar43    etc8   etc60   bar33   bar59
   0.28    0.17    0.07    0.06    0.05    0.05    0.05    0.05    0.04    0.03
  per15   per60   degc1   etc65   bar67 degc209    per5    per23   etc70 
   0.03    0.03    0.02    0.02    0.01    0.01    0.01     0.01    0.01 

And this is the output from 2:

> stabsel(php.glmb, cutoff=0.75, q=10)
        Stability Selection with unimodality assumption

Selected base-learners:
degc1  per5 per15 per45 per60  etc8 etc60 etc65 etc70 
   20    43    44    52    54    60    72    74    76 

Selection probabilities:
(Intercept)       bar25       bar26       bar28       bar29       bar32       bar42 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
      bar43       bar45       bar46       bar49       bar50       bar59       bar60 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
      bar62       bar63       bar66       degc2       degc3      degc16     degc109 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
    degc111     degc147     degc154     degc155     degc158     degc181     degc183 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
    degc204     degc205     degc206     degc209     degc229     degc231     degc238 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
    degc255     degc256     degc257     degc260       per21       per24       per34 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
      per40       per42       per43       per58       per61       per63        etc5 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
       etc6        etc7       etc11       etc15       etc26       etc30       etc32 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
      etc33       etc41       etc45       etc47       etc48       etc59       etc64 
       0.00        0.00        0.00        0.00        0.00        0.00        0.00 
      etc69       bar67       bar33       per23       per60       degc1        per5 
       0.00        0.15        0.18        0.71        0.96        1.00        1.00 
      per15       per45        etc8       etc60       etc65       etc70 
       1.00        1.00        1.00        1.00        1.00        1.00 

So the important variables are almost completely disjoint: method 1 says degc238, etc48, etc7 and per45 are the most important (have the highest selection probabilities), while method 2 says etc70, etc65, etc60, etc8, per60 and so on.

What can be the reason for this? I should also mention that there's a significant amount of collinearity in this dataset; several predictors have univariate correlations of 90%+ with the response and with each other. Could this be impacting the result?

Answer 1

You are asking two completely different questions when you look at the selection frequency within one data set (as returned by the model summary) and the stability selection results:

1. Selection frequencies of the base-learners give you the answer how many updates were needed for this variable to reach its current value. This has little to nothing to do with its importance. It is influenced by the iteration in which the variable first was selected (the earlier the bigger usually the contribution, at least relative to later iterations), the effect size used for the update (i.e., the regression coefficient), etc. More importantly, from your output it seems that you did not cross-validate (i.e., tune) your model. If you do this (using cvrisk) results might change substantially.

2. Stability selection tells you on how many subsets (of size 1/2) of the data a certain variable was selected among the first q distinct selected variables (10 in your case). This gives you a notion of how stable the model is with respect to changes of the data. It also allows you to detect variables which are influential irrespective of these changes.

Collinearity should not be a (big) issue as boosting is a regularization approach and thus is able to deal with it. The output of stabsel also shows that we have a very stable model (which is contradicting the collinearity issue). 8 variables were selected in each of the subsets, 1 was selected in 96% of the subsets and only three other variables were selected among the first 10.

All of the variables selected by stabsel were also found in your original boosting model. So I do not see that the evaluation methods are contradictory; they only answer different questions as layed out above.