Which is the best classifier and with what performance measures?

I tried to implement a Classifier comparison like in the scikit-learn for text classification.

I used an 81 instances as a training sample and a 46 instances as a test sample. I tried several situation with three classifier the K-Nearest Neighbors, the Random Forest Classifier and the Decision Tree Classifier. To measures theirs performance I used different performance measures.

Edit 2: As PhilippPro mentioned in his comment, there was indeed a problem in my experiment, I repeated the experiment by considering an 81 samples that was found by an unsupervised classification as a first train model and a 13 samples labeled manually as a second train sample to test a 46 samples. In the end this is what I got.

So my question is, could I say with certitude that the best classifier in this situation is the Decision Tree Classifier with an F1-score of 82.02%.

Edit 1: Like in the comment of halilpazarlama I considered the idea of Cross Validation which i found in the [Cross_validation_sklearn] for only the 46 samples (I was wrong before) with the code below:

# Cross Validation
def cross_validation_score(model):
text_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', model),])
#text_clf = text_clf.fit(X_train, Y_train)
#Y_pred = text_clf.predict(X_test)
return cross_validation.cross_val_score(text_clf, X_test, Y_pred, cv=7)

print('\n################Cross Validation Score###########\n')
for i in models:
scores=cross_validation_score(models[i])
print(i + ':\n' + str(scores))
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
print('#######################################\n')


and this is the result that I got for the three classifier:

K-Nearest Neighbors: [ 87.5 37.5 62.5 71.42857143 80. 60. 40. ] Accuracy: 62.70 (+/- 35.04)

#

Random Forest Classifier: [ 87.5 87.5 87.5 100. 100. 100. 60. ] Accuracy: 88.93 (+/- 26.30)

#

Decision Tree Classifier: [ 87.5 87.5 87.5 100. 100. 100. 100. ] Accuracy: 94.64 (+/- 12.37)

#

Sorry but I am a little confused, does it mean that I got an accuracy of 95% with the Decision Tree Classifier for a supervised classification. What did change exactly?

Thanks

• seems like it, yes. but why not cross validation? – jeff Jul 2 '16 at 12:30
• I think this paper addresses the full scope of issues implicit in your question Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fernandez-Delgado, Cernadas and Barro, available here ... jmlr.org/papers/volume15/delgado14a/delgado14a.pdf – Mike Hunter Jul 2 '16 at 12:39
• Do I understand correctly that you used different training/test samples in your trials (which would be valid), and got de facto equal results for many of those? If yes I'd be careful about those numbers: there should be a very small chance of getting de facto equal results, except e.g. many of your samples being exactly equal, so there might be something wrong underneath. – geekoverdose Jul 8 '16 at 15:48

Your table confuses me, but if the decision-tree classifier indeed got higher accuracy (73.91%) for the test set, when fit on the training set, than the other two models did, then yes, it's the best-performing classifier. But you don't know with certainty (certitude) that this classifier will do best in the future. Your training set and test set are finite, so the accuracy scores (and $F_1$ scores, and all the other performance measures) are only estimates.