How to compare 2 classifiers after testing trained model in Weka?

Question

I'm new to Weka (and machine learning) so this question would be a bit silly.

So I have 2 models built using J48 and RandomForest (both run with 10-fold cross-validation mode) on a 40,000-tuple training set. I also have 6 different smaller test sets, each has around 8000 tuples.

Now I want to compare how these 2 classification algorithms perform after testing their models (I have the predictions reported).

The problem is, I don't have a good understanding of the reported predictions so I don't know how to compare the prediction of each to see which one is better in terms of accuracy.

Answer 1

For classifier comparison, you can usually set up a paired experiment. That is, look at predictions of the same case by both models instead of at overall precision of the models.

McNemar's test in Wikipedia (or here) should get you started.

Answer 2

What determines the performance of your classification systems? Success rate? AUC? F-measure? and so on. This wiki page is reasonable for an introduction on all of that: http://en.wikipedia.org/wiki/Receiver_operating_characteristic.

WEKA reports most of these measures quite clearly (in the summary output), and you can use a ThresholdCurve to plot ROC curves.

Once you've decided that, confidence intervals and the paired t-test are commonly used.

These slides aren't bad also: http://web.engr.oregonstate.edu/~tgd/classes/534/slides/part13.pdf