2
$\begingroup$

I was trying the filterVarImp function in caret package. While the outcome was categorical we get ROC as the importance measure. My question is: How is the function doing it? Is it fitting a model and then predicting probabilities to compare with target to give AUC or comparing itself directly? How is categorical predictor handled here?

$\endgroup$

closed as off-topic by gung, Michael Chernick, John, Nick Cox, Peter Flom Jan 10 '17 at 12:06

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – gung, John, Nick Cox, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ Why not read the documentation, or even examine the underlying code? At any rate, questions about how R (R functions) works are off topic here. $\endgroup$ – gung Jan 4 '17 at 3:15
2
$\begingroup$

The documentation for the function can be accessed using:

?filterVarImp

It states the following:

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance.

For categorical predictors, one can imagine the function is using a similar approach. It just needs to have some kind of order to apply a series of cutoffs. This could be obtained using factor levels or alphabetical ordering.

Here is an example:

testDf <- data.frame(x1 = c(1, 2, 1, 2, 3, 54), 
                     x2 = c("a", "b", "a", "b", "a", "b"), 
                     x3 = c("a", "b", "c", "a", "b", "c"),
                     y = c("N", "N", "N", "Y", "Y", "Y"))
filterVarImp(x = testDf[, -4], 
             y = testDf$y)

That gives the following output:

           N         Y
x1 0.9444444 0.9444444
x2 0.6666667 0.6666667
x3 0.5000000 0.5000000

But I am seeing that this function can be misleading if the output levels are flipped. For example if you flip the output variable levels like this:

testDf <- data.frame(x1 = c(1, 2, 1, 2, 3, 54), 
                     x2 = c("a", "b", "a", "b", "a", "b"), 
                     x3 = c("a", "b", "c", "a", "b", "c"),
                     y = c("Y", "Y", "Y", "N", "N", "N")
                     )
filterVarImp(x = testDf[, -4], 
             y = testDf$y)

the output becomes:

            N          Y
x1 0.05555556 0.05555556
x2 0.33333333 0.33333333
x3 0.50000000 0.50000000

AUCs under 0.5 need to be subtracted from 1 to get the true AUC.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.