0
$\begingroup$

In a linear SVM model, inside caret I would like to get the variable importance after recursive feature elimination, so according to the documentation:

library(caret)
library(mlbench)
library(Hmisc)
library(randomForest)
n <- 100
p <- 40
sigma <- 1
set.seed(1)
sim <- mlbench.friedman1(n, sd = sigma)
colnames(sim$x) <- c(paste("real", 1:5, sep = ""),
                         paste("bogus", 1:5, sep = ""))
    bogus <- matrix(rnorm(n * p), nrow = n)
    colnames(bogus) <- paste("bogus", 5+(1:ncol(bogus)), sep = "")
    x <- cbind(sim$x, bogus)
y <- sim$y



normalization <- preProcess(x)
x <- predict(normalization, x)
x <- as.data.frame(x)
subsets <- c(1:5, 10, 15, 20, 25)
set.seed(10)
ctrl <- rfeControl(functions = caretFuncs,
                   method = "repeatedcv",
                   repeats = 5,
                   verbose = FALSE)

lmProfile <- rfe(x, y,
                 sizes = subsets,
                 rfeControl = ctrl)

lmProfile

The output of lmProfile shows the most 5 important variables, and predictors(lmProfile) will show me all the important (retained) predictors after recursive feature elimination. In the documentation I can read this sentence:

... At first this may seem like a disadvantage, but it does provide a more probabilistic assessment of predictor importance than a ranking based on a single fixed data set. At the end of the algorithm, a consensus ranking can be used to determine the best predictors to retain.

Could anyone please let me know how I can get these probabilistic numbers for each retained variable? In other words I would like to report my important variables not only by showing their ranks, but in a more quantitative way.

$\endgroup$
1
$\begingroup$

You will need to look at the results in lmProfile$variables. That has the variable importance values per resample and subset size. For your example, using random forest (not linear models), I used rfFuncs which is faster since it doesn't tune over mtry:

set.seed(10)
ctrl <- rfeControl(functions = rfFuncs,
                   method = "repeatedcv",
                   repeats = 5,
                   verbose = FALSE)
set.seed(1)
rfProfile <- rfe(x, y,
                 sizes = subsets,
                 rfeControl = ctrl)

The data look like:

> head(rfProfile$variables)
                       Overall     var Variables    Resample
selectedVars.real4   18.721549   real4        50 Fold01.Rep1
selectedVars.real2   10.917085   real2        50 Fold01.Rep1
selectedVars.real5   10.305418   real5        50 Fold01.Rep1
selectedVars.real1    8.813745   real1        50 Fold01.Rep1
selectedVars.bogus17  3.932762 bogus17        50 Fold01.Rep1
selectedVars.bogus44  2.384077 bogus44        50 Fold01.Rep1

You can get the rate that each predictor was selected at each iteration:

library(plyr)
selected <- ddply(rfProfile$variables, 
                      .variables = c("Variables"),
                      function(x) as.data.frame(table(x$var)))
selected$Prob <- selected$Freq/length(rfProfile$control$index)

For example:

> subset(selected, Variables == 5)
   Variables    Var1 Freq Prob
14         5 bogus14    4 0.08
15         5 bogus17   35 0.70
16         5  bogus2    1 0.02
17         5 bogus26    1 0.02
18         5 bogus44    7 0.14
19         5   real1   50 1.00
20         5   real2   50 1.00
21         5   real3    2 0.04
22         5   real4   50 1.00
23         5   real5   50 1.00

These are the variables that were selected when 5 predictors are retained over the 50 resamples. real1, real2, real4 and real5 were all selected each time.

You can also get summary statistics for the Overall variable importance score for each predictor also.

Max

$\endgroup$
  • $\begingroup$ Thank you. Could you also please describe what I should do when I have binary responses? same code with binary y will give me only 1 s as the probability $\endgroup$ – Arman Oct 8 '14 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.