filterVarImp - Caret Package [closed]

Question

I was trying the filterVarImp function in caret package. While the outcome was categorical we get ROC as the importance measure. My question is: How is the function doing it? Is it fitting a model and then predicting probabilities to compare with target to give AUC or comparing itself directly? How is categorical predictor handled here?

Answer 1

The documentation for the function can be accessed using:

?filterVarImp

It states the following:

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance.

For categorical predictors, one can imagine the function is using a similar approach. It just needs to have some kind of order to apply a series of cutoffs. This could be obtained using factor levels or alphabetical ordering.

Here is an example:

testDf <- data.frame(x1 = c(1, 2, 1, 2, 3, 54), 
                     x2 = c("a", "b", "a", "b", "a", "b"), 
                     x3 = c("a", "b", "c", "a", "b", "c"),
                     y = c("N", "N", "N", "Y", "Y", "Y"))
filterVarImp(x = testDf[, -4], 
             y = testDf$y)

That gives the following output:

           N         Y
x1 0.9444444 0.9444444
x2 0.6666667 0.6666667
x3 0.5000000 0.5000000

But I am seeing that this function can be misleading if the output levels are flipped. For example if you flip the output variable levels like this:

testDf <- data.frame(x1 = c(1, 2, 1, 2, 3, 54), 
                     x2 = c("a", "b", "a", "b", "a", "b"), 
                     x3 = c("a", "b", "c", "a", "b", "c"),
                     y = c("Y", "Y", "Y", "N", "N", "N")
                     )
filterVarImp(x = testDf[, -4], 
             y = testDf$y)

the output becomes:

            N          Y
x1 0.05555556 0.05555556
x2 0.33333333 0.33333333
x3 0.50000000 0.50000000

AUCs under 0.5 need to be subtracted from 1 to get the true AUC.