What is a good resource that includes a comparison of the pros and cons of different classifiers?

Question

What is the best out-of-the-box 2-class classifier? Yes, I guess that's the million dollar question, and yes, I'm aware of the no free lunch theorem, and I've also read the previous questions:

• What is the best out-of-the-box 2-class classifier for your application?
• and Worst classifier

Still, I'm interested in reading more on the subject.

What is a good source of information that includes a general comparison of the characteristics, advantage, and features of different classifiers?

Answer 1

The ESL, as already mentioned by Peter Flom, is an excellent suggestion (note that my link is to the author's homepage where the book can be obtained as a pdf-file for free). Let me add a couple of more specific things to look for in the book:

• Table 10.1 (page 351) gives the authors assessment of certain characteristics of Neural Nets, SVM, Trees, MARS, and k-NN kernels, which somehow appear to be the methods the authors want to include in a list of "off-the-shelf" methods.
• Chapter 10 treats boosting, which I found missing in the list of methods in the poll cited by the OP. Gradient boosting seems to be one of the better performing methods in a number of examples.
• Chapter 9 treats generalized additive models (GAMs), which adds to the logistic regression model (top ranked in the poll) the flexibility of non-linear additive effects of the predictors. GAMs would not be nearly as easy to use as logistic regression with all the smoothing parameters that have to be chosen if it wasn't for nice implementations like the one in the R package mgcv.

Add to the book the Machine Learning Task View for R, which gives some impression of what the many machine learning packages can actually do, though there is no real comparison. For Python users I imagine that scikit.learn is a good place to look. How much "out-of-the-box" or "off-the-shelf" a method is, is very much determined by how well the implementation deals with automatic adaptation to the data situation versus leaving the detailed tuning to the user. In my mind, mgcv for R is a good example that makes the fitting of a reasonably good generalized additive model really easy and basically without any need for the user to "hand-tune" anything.

Answer 2

The resources listed by others are all certainly useful, but I'll chime in and add the following: the "best" classifier is likely to be context and data specific. In a recent foray into assessing different binary classifiers I found a Boosted Regression Tree to work consistently better than other methods I had access to. The key thing for me was learning how to use Orange data mining tools. They have some great documentation to get started on exploring these methods with your data. For example, here is a short Python script I wrote to assess the quality of multiple classifiers across multiple measures of accuracy using k-fold cross validation.

import orange, orngTest, orngStat, orngTree , orngEnsemble, orngSVM, orngLR
import numpy as np

data = orange.ExampleTable("performance_orange_2.tab")
bayes = orange.BayesLearner(name="Naive Bayes")
svm = orngSVM.SVMLearner(name="SVM")
tree = orngTree.TreeLearner(mForPruning=2, name="Regression Tree")
bs = orngEnsemble.BoostedLearner(tree, name="Boosted Tree")
bg = orngEnsemble.BaggedLearner(tree, name="Bagged Tree")
forest = orngEnsemble.RandomForestLearner(trees=100, name="Random Forest")
learners = [bayes, svm, tree, bs, bg, forest]
results = orngTest.crossValidation(learners, data, folds=10)
cm = orngStat.computeConfusionMatrices(results,
                             classIndex=data.domain.classVar.values.index('1'))

stat = (('ClsAcc', 'CA(results)'),
        ('Sens', 'sens(cm)'),
        ('Spec', 'spec(cm)'),
        ('AUC', 'AUC(results)'),
        ('Info', 'IS(results)'),
        ('Brier', 'BrierScore(results)'))
scores = [eval("orngStat." + s[1]) for s in stat]
print "Learner        " + "".join(["%-9s" % s[0] for s in stat])
print "-----------------------------------------------------------------"
for (i, L) in enumerate(learners):
    print "%-15s " % L.name + "".join(["%5.3f   " % s[i] for s in scores])

print "\n\n"
measure = orngEnsemble.MeasureAttribute_randomForests(trees=100)
print "Random Forest Variable Importance"
print "---------------------------------"
imps = measure.importances(data)
for i,imp in enumerate(imps):
    print "%-20s %6.2f" % (data.domain.attributes[i].name, imp)

print '\n\n'
print 'Predictions on new data...'
bs_classifier = bs(data)
new_data = orange.ExampleTable('performance_orange_new.tab')
for obs in new_data:
    print bs_classifier(obs, orange.GetBoth)

When I run this code on my data I get output like

In [1]: %run binary_predict.py
Learner        ClsAcc   Sens     Spec     AUC      Info     Brier
-----------------------------------------------------------------
Naive Bayes     0.556   0.444   0.643   0.756   0.516   0.613
SVM             0.611   0.667   0.714   0.851   0.264   0.582
Regression Tree 0.736   0.778   0.786   0.836   0.945   0.527
Boosted Tree    0.778   0.778   0.857   0.911   1.074   0.444
Bagged Tree     0.653   0.667   0.786   0.816   0.564   0.547
Random Forest   0.736   0.667   0.929   0.940   0.455   0.512


Random Forest Variable Importance
---------------------------------
Mileage            2.34
Trade_Area_QI      2.82
Site_Score         8.76

There is a lot more you can do with the Orange objects to introspect performance and make comparisons. I found this package to be extremely helpful in writing a small amount of code to actually apply methods to my data with a consistent API and problem abstraction (i.e., I did not need to use six different packages from six different authors, each with their own approach to API design and documentation, etc).

Answer 3

The book The Elements of Statistical Learning has a lot of information on this.

Answer 4

Other resources I found regarding this (free PDF available):

• The book Machine Learning, Neural and Statistical Classification, edited by D. Michie, D.J. Spiegelhalter and C.C. Taylor
• Rich Caruana and Alexandru Niculescu-mizil, An Empirical Comparison of Supervised Learning Algorithms

Answer 5

According to this exhaustive recent study (evaluation of 179 classifiers on 121 datasets), the best classifiers are random forests followed by support vector machines.