Extending 2-class models to multi-class problems

Question

This paper on Adaboost gives some suggestions and code (page 17) for extending 2-class models to K-class problems. I would like to generalize this code, such that I can easily plug in different 2-class models and compare the results. Because most classification models have a formula interface and a predict method, some of this should be relatively easy. Unfortunately, I haven't found a standard way of extracting class probabilities from 2-class models, so each model will require some custom code.

Here's a function I wrote to break up a K-class problem into 2-class problems, and return K models:

oneVsAll <- function(X,Y,FUN,...) {
    models <- lapply(unique(Y), function(x) {
        name <- as.character(x)
        .Target <- factor(ifelse(Y==name,name,'other'), levels=c(name, 'other'))
        dat <- data.frame(.Target, X)
        model <- FUN(.Target~., data=dat, ...)
        return(model)
    })
    names(models) <- unique(Y)
    info <- list(X=X, Y=Y, classes=unique(Y))
    out <- list(models=models, info=info)
    class(out) <- 'oneVsAll'
    return(out)
}

Here's a prediction method I wrote to iterate over each model and make predictions:

predict.oneVsAll <- function(object, newX=object$info$X, ...) {
    stopifnot(class(object)=='oneVsAll')
    lapply(object$models, function(x) {
        predict(x, newX, ...)
    })
}

And finally, here's a function to turn normalize a data.frame of predicted probabilities and classify the cases. Note that it is up to you to construct the K-column data.frame of probabilities from each model, as there is not a unified way to extract class probabilities from a 2-class model:

classify <- function(dat) {
    out <- dat/rowSums(dat)
    out$Class <- apply(dat, 1, function(x) names(dat)[which.max(x)])
    out
}

Here's an example using adaboost:

library(ada)
library(caret) 
X <- iris[,-5]
Y <- iris[,5]
myModels <- oneVsAll(X, Y, ada)
preds <- predict(myModels, X, type='probs')
preds <- data.frame(lapply(preds, function(x) x[,2])) #Make a data.frame of probs
preds <- classify(preds)
>confusionMatrix(preds$Class, Y)
Confusion Matrix and Statistics

            Reference
Prediction   setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         47         2
  virginica       0          3        48

Here is an example using lda (I know lda can handle multiple classes, but this is just an example):

library(MASS)
myModels <- oneVsAll(X, Y, lda)
preds <- predict(myModels, X)
preds <- data.frame(lapply(preds, function(x) x[[2]][,1])) #Make a data.frame of probs
preds <- classify(preds)
>confusionMatrix(preds$Class, Y)
Confusion Matrix and Statistics

            Reference
Prediction   setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         39         5
  virginica       0         11        45

These functions should work for any 2-class model with a formula interface and a predict method. Note that you have to manually split up the X and Y components, which is a little ugly, but writing a formula interface is beyond me at the moment.

Does this approach make sense to everyone? Is there any way I can improve it, or is there an existing package to solve this issue?

Answer 1

One way to improve is to use "weighted all pairs" approach which is supposedly better than "one against all" while still scalable.

As for existing packages, glmnet supports (regularized) multinomial logit which can be used as a multi-class classifier.