This paper on Adaboost gives some suggestions and code (page 17) for extending 2-class models to K-class problems. I would like to generalize this code, such that I can easily plug in different 2-class models and compare the results. Because most classification models have a formula interface and a predict method, some of this should be relatively easy. Unfortunately, I haven't found a standard way of extracting class probabilities from 2-class models, so each model will require some custom code.

Here's a function I wrote to break up a K-class problem into 2-class problems, and return K models:

oneVsAll <- function(X,Y,FUN,...) {
    models <- lapply(unique(Y), function(x) {
        name <- as.character(x)
        .Target <- factor(ifelse(Y==name,name,'other'), levels=c(name, 'other'))
        dat <- data.frame(.Target, X)
        model <- FUN(.Target~., data=dat, ...)
    names(models) <- unique(Y)
    info <- list(X=X, Y=Y, classes=unique(Y))
    out <- list(models=models, info=info)
    class(out) <- 'oneVsAll'

Here's a prediction method I wrote to iterate over each model and make predictions:

predict.oneVsAll <- function(object, newX=object$info$X, ...) {
    lapply(object$models, function(x) {
        predict(x, newX, ...)

And finally, here's a function to turn normalize a data.frame of predicted probabilities and classify the cases. Note that it is up to you to construct the K-column data.frame of probabilities from each model, as there is not a unified way to extract class probabilities from a 2-class model:

classify <- function(dat) {
    out <- dat/rowSums(dat)
    out$Class <- apply(dat, 1, function(x) names(dat)[which.max(x)])

Here's an example using adaboost:

X <- iris[,-5]
Y <- iris[,5]
myModels <- oneVsAll(X, Y, ada)
preds <- predict(myModels, X, type='probs')
preds <- data.frame(lapply(preds, function(x) x[,2])) #Make a data.frame of probs
preds <- classify(preds)
>confusionMatrix(preds$Class, Y)
Confusion Matrix and Statistics

Prediction   setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         47         2
  virginica       0          3        48

Here is an example using lda (I know lda can handle multiple classes, but this is just an example):

myModels <- oneVsAll(X, Y, lda)
preds <- predict(myModels, X)
preds <- data.frame(lapply(preds, function(x) x[[2]][,1])) #Make a data.frame of probs
preds <- classify(preds)
>confusionMatrix(preds$Class, Y)
Confusion Matrix and Statistics

Prediction   setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         39         5
  virginica       0         11        45

These functions should work for any 2-class model with a formula interface and a predict method. Note that you have to manually split up the X and Y components, which is a little ugly, but writing a formula interface is beyond me at the moment.

Does this approach make sense to everyone? Is there any way I can improve it, or is there an existing package to solve this issue?

  • 2
    $\begingroup$ Wow, until you asked and I looked, I'd have been sure that some package (like car, or one of the *lab packages) would have provided a function like yours. Sorry I can't help. I've read a bit about how k-way SVM works and it seems like it was more complicated than I'd have thought. $\endgroup$
    – Wayne
    Feb 14, 2012 at 22:06
  • 1
    $\begingroup$ @Wayne: Me too! I was certain there'd be some general function that would do this, provided the model has a predict method. $\endgroup$
    – Zach
    Feb 15, 2012 at 14:18

1 Answer 1


One way to improve is to use "weighted all pairs" approach which is supposedly better than "one against all" while still scalable.

As for existing packages, glmnet supports (regularized) multinomial logit which can be used as a multi-class classifier.

  • $\begingroup$ I'm aware of the many packages in R that support multi-class classification (such as glmnet, random forests, kernlab, rpart, nnet, etc.). I'm more curious about extending binary classification packages (e.g. gbm) to multiclass problems. I'll look into "weighted all pairs." $\endgroup$
    – Zach
    Feb 15, 2012 at 14:16
  • $\begingroup$ Also, it's interesting that glmnet includes a multinomial loss function. I wonder if this loss function could be used in other algorithms in R, such as ada or gbm? $\endgroup$
    – Zach
    Mar 22, 2012 at 17:59
  • $\begingroup$ Yes, some methods can be extended to support multinomial loss function. For example, kernel logistic regression is extended that way here: books.nips.cc/papers/files/nips14/AA13.pdf As far as know ada is "reserved" for a specific (exponential) loss function, but one could extend another boosting-based method to support the multinomial loss function - e.g. see page 360 of The Elements of Statistical Learning for details on multi-class GBM - K binary trees are built for each boosting iteration where K is the number of classes (just one tree per iteration is needed in binary case). $\endgroup$
    – Yevgeny
    Mar 23, 2012 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.