Thanks for taking the time to read this I'm new to Machine Learning and so am going through a Kaggle competition to help me improve but I have a question. How can I compare different classifiers?? My Python isn't as neat as I would like it to be but i think it's correct. Please let me know if I'm doing something strange

train = pca(train)
cfr = KNeighborsClassifier(n_neighbors=neighbours, algorithm="kd_tree")
cv = cross_validation.KFold(len(train), n_folds=10, indices=True)

results = []
i = 0
count =0
for traincv, testcv in cv:
    ClassPred = cfr.fit(train[traincv], target[traincv]).predict(train[testcv]) 
    for j in range(0,(len(train)/10)):
        labelindex = testcv[j]
        if (ClassPred[j] == target[labelindex]):
            i = i+1
    accuracy = (i/2000.0)*100
    count = count + 1
    print "accuracy for fold", count, " : ", accuracy,"%"
    print "time after fold", count, " : ", elspasedfold,"%"

elapsed = (time.clock() - start)
#print out the mean of the cross-validated results
print "Results for RBF, c = 10.0, gamma=0.1 \n" + str(np.array(results).mean())
print "Time taken is %ds" % elapsed`.

As you can see I'm doing 10 fold cross validation on the training data and hopefully producing an 'accuracy' value out of the other end. My question here is how can I compare different classifiers?? If I choose cfr as a different classifier (e.g. Random Forest) and get a value for that how do i statistically compare the two?

My initial thoughts are to use a two-tailed t-test on the values for each fold (so 10 values per classifier) instead of just taking using the one value it currently outputs, which I think is an average of all the folds combined. I would need to then get a p-value to see if the differences are significant. I am not sure how I would go about implementing this however or if this is the correct thing to do. My stats is a little patchy (Which I'm working on) but any help anyone can give me would be much appreciated.

So to clarify, using python and sci-kit what would be the best way to compare the performance of different classifiers on my data? Thanks!

EDIT: The tutorial assessment criteria is that 10 fold cross validation must be used.


1 Answer 1


As you loop over the 10 folds, each fold returns an accuracy. That accuracy varies a little bit, depending on how you slice your data for each fold. The range of accuracies that you obtain represents the range of variability in the performance of your model that you might expect to see, if you brought that model to bear upon a brand new set of test data.

From the 10 accuracies that you obtain, for each classifier, you can calculate a mean and a corrected sample standard deviation. You would like to know, for two classifiers with two different average accuracies, whether those differences are "significant", i.e., whether the differences are meaningfully different, or are simply due to the same random statistical fluctuations that account for the same fold-to-fold random variation that you have already observed in the first place.

If you have an average accuracy $A_{1}$ and standard deviation $\sigma_{1}$ for classifier number 1, and the same for some other classifier number 2, you can estimate whether the difference in their relative performance is meaningfully different from zero by calculating $$\Delta_{12} = \frac{A_{1} - A_{2}}{\sqrt{\sigma_{1}^{2} + \sigma_{2}^{2}}}$$ This quantity can be interpreted effectively as a kind of Z score. If the score is large (a value greater than 3 standard deviations is a common choice of cutoff) you may declare the performance of the two classifiers to be significantly different, if not, they are essentially equivalent.

  • $\begingroup$ Thanks, this looks like exactly what I needed. According to wikipedia "z requires the population mean and the population standard deviation, not the sample mean or sample deviation." Could I say that I have the population mean/sd since cross validation should be using the entire test set? I'm concerned I should be using the t-test instead. Thanks! $\endgroup$
    – Azureaus
    Dec 17, 2013 at 10:42
  • $\begingroup$ Good catch. What the wikipedia page actually says is, "The z-score is only defined if one knows the population parameters; if one only has a sample set, then the analogous computation with sample mean and sample standard deviation yields the Student's t-statistic." I used the phrase, "can be interpreted effectively as a kind of" in my answer because I wan't confident what was the best terminology to use. You are definitely taking samples, so t-statistic is probably the better terminology to describe what I had recommended doing, but other than that, my basic advice does not change. $\endgroup$
    – stachyra
    Dec 17, 2013 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.