Probabilistic importance value for Caret linear SVM classifier

In a linear SVM model, inside caret I would like to get the variable importance after recursive feature elimination, so according to the documentation:

library(caret)
library(mlbench)
library(Hmisc)
library(randomForest)
n <- 100
p <- 40
sigma <- 1
set.seed(1)
sim <- mlbench.friedman1(n, sd = sigma)
colnames(sim$x) <- c(paste("real", 1:5, sep = ""), paste("bogus", 1:5, sep = "")) bogus <- matrix(rnorm(n * p), nrow = n) colnames(bogus) <- paste("bogus", 5+(1:ncol(bogus)), sep = "") x <- cbind(sim$x, bogus)
y <- sim$y normalization <- preProcess(x) x <- predict(normalization, x) x <- as.data.frame(x) subsets <- c(1:5, 10, 15, 20, 25) set.seed(10) ctrl <- rfeControl(functions = caretFuncs, method = "repeatedcv", repeats = 5, verbose = FALSE) lmProfile <- rfe(x, y, sizes = subsets, rfeControl = ctrl) lmProfile The output of lmProfile shows the most 5 important variables, and predictors(lmProfile) will show me all the important (retained) predictors after recursive feature elimination. In the documentation I can read this sentence: ... At first this may seem like a disadvantage, but it does provide a more probabilistic assessment of predictor importance than a ranking based on a single fixed data set. At the end of the algorithm, a consensus ranking can be used to determine the best predictors to retain. Could anyone please let me know how I can get these probabilistic numbers for each retained variable? In other words I would like to report my important variables not only by showing their ranks, but in a more quantitative way. 1 Answer You will need to look at the results in lmProfile$variables. That has the variable importance values per resample and subset size. For your example, using random forest (not linear models), I used rfFuncs which is faster since it doesn't tune over mtry:

set.seed(10)
ctrl <- rfeControl(functions = rfFuncs,
method = "repeatedcv",
repeats = 5,
verbose = FALSE)
set.seed(1)
rfProfile <- rfe(x, y,
sizes = subsets,
rfeControl = ctrl)

The data look like:

> head(rfProfile$variables) Overall var Variables Resample selectedVars.real4 18.721549 real4 50 Fold01.Rep1 selectedVars.real2 10.917085 real2 50 Fold01.Rep1 selectedVars.real5 10.305418 real5 50 Fold01.Rep1 selectedVars.real1 8.813745 real1 50 Fold01.Rep1 selectedVars.bogus17 3.932762 bogus17 50 Fold01.Rep1 selectedVars.bogus44 2.384077 bogus44 50 Fold01.Rep1 You can get the rate that each predictor was selected at each iteration: library(plyr) selected <- ddply(rfProfile$variables,
.variables = c("Variables"),
function(x) as.data.frame(table(x$var))) selected$Prob <- selected$Freq/length(rfProfile$control\$index)

For example:

> subset(selected, Variables == 5)
Variables    Var1 Freq Prob
14         5 bogus14    4 0.08
15         5 bogus17   35 0.70
16         5  bogus2    1 0.02
17         5 bogus26    1 0.02
18         5 bogus44    7 0.14
19         5   real1   50 1.00
20         5   real2   50 1.00
21         5   real3    2 0.04
22         5   real4   50 1.00
23         5   real5   50 1.00

These are the variables that were selected when 5 predictors are retained over the 50 resamples. real1, real2, real4 and real5 were all selected each time.

You can also get summary statistics for the Overall variable importance score for each predictor also.

Max

• Thank you. Could you also please describe what I should do when I have binary responses? same code with binary y will give me only 1 s as the probability – Arman Oct 8 '14 at 9:34