Minimizing Single-Class Classification Error in R Trees

Question

I'm running tree models in R to help define rules for predicting a binary outcome (0 or 1, of course). I understand mostly how to algorithm works, but I'm in a position where I don't really care about the overall classification error, but rather I'd like to minimize the error of finding when the outcome is 1. Is there a way to point the algorithm to minimize the classification error of only one class?

I'm using rpart because I'm partial to its tree-drawing capabilities but I'm open to other packages.

Thanks!

Answer 1

There are two simple options you could try for your problem: using sample weights and down/upsampling, which will both change how your model fits to your data.

By using sample weights, you weight e.g. the error for certain samples more (for less prominent classes), or less (for prominent classes). By using downsampling, you subset the more prominent classes to equal size as the smallest class, and by upsampling, you analogically artificially create more samples of the less prominent classes to have equal size to the biggest class. The latter can be tricky, but with trees and e.g. including the same sample multiple times in the data you obtain a similar effect to using sample weights (e.g. the same sample being classified wrong twice would mean increased error).

Here's a small example in R, using the mtcars data for demonstration (not using rpart directly, but wrapped in the caret package):

library(caret)
# "6" cyl are underrepresented
table(mtcars[,2])
#  4  6  8 
# 11  7 14 

# train plain model
set.seed(123456)
m <- train(mtcars[,c(1,3)], factor(mtcars[,2]), method = 'rpart', metric = 'Kappa', tuneGrid = expand.grid(cp = 0.1), trControl = trainControl(method = 'LOOCV', savePredictions = T))
# prediction for "6" is bad
confusionMatrix(m$pred$pred, m$pred$obs)
# [...]
#                      Class: 4 Class: 6 Class: 8
# Sensitivity            1.0000   0.0000   1.0000
# Specificity            0.7143   1.0000   0.9444
# [...]    

# option 1: sample weights - you *will need* to adapt weights to fit your class prevalence
set.seed(123456)
m2 <- train(mtcars[,c(1,3)], factor(mtcars[,2]), method = 'rpart', metric = 'Kappa', tuneGrid = expand.grid(cp = 0.1), trControl = trainControl(method = 'LOOCV', savePredictions = T), weights = ifelse(mtcars[,2]==6,2,1))
confusionMatrix(m2$pred$pred, m2$pred$obs)
# [...]
#                      Class: 4 Class: 6 Class: 8
# Sensitivity            0.0000   0.8571   1.0000
# Specificity            1.0000   0.5600   0.9444
# [...]  

# option 2: upsampling
d <- upSample(x = mtcars[,c(1,3)], factor(mtcars[,2]), yname = 'cyl')
table(d$cyl)
#  4  6  8 
# 14 14 14
set.seed(123456)
m3 <- train(d[,c(1,2)], d[,3], method = 'rpart', metric = 'Kappa', tuneGrid = expand.grid(cp = 0.1), trControl = trainControl(method = 'LOOCV', savePredictions = T))
confusionMatrix(m3$pred$pred, m3$pred$obs)
# [...]
#                      Class: 4 Class: 6 Class: 8
# Sensitivity            0.8571   0.8571   1.0000
# Specificity            0.9286   0.9286   1.0000
# [...]  

In case you really care for one specific class A most, you can of course emphasize this class further, using even higher weights or oversampling. You could also think about simplifying your problem to binary classification (A vs rest) - this might have a positive impact on results too.

PS: I've pretty much skipped model tuning in the snippet - in case you use it for your purpose be sure to include it again.