1
$\begingroup$

I tried to implement a Classifier comparison like in the scikit-learn for text classification.

I used an 81 instances as a training sample and a 46 instances as a test sample. I tried several situation with three classifier the K-Nearest Neighbors, the Random Forest Classifier and the Decision Tree Classifier. To measures theirs performance I used different performance measures.

Edit 2: As PhilippPro mentioned in his comment, there was indeed a problem in my experiment, I repeated the experiment by considering an 81 samples that was found by an unsupervised classification as a first train model and a 13 samples labeled manually as a second train sample to test a 46 samples. In the end this is what I got. Classifier Comparison

So my question is, could I say with certitude that the best classifier in this situation is the Decision Tree Classifier with an F1-score of 82.02%.

Edit 1: Like in the comment of halilpazarlama I considered the idea of Cross Validation which i found in the [Cross_validation_sklearn] for only the 46 samples (I was wrong before) with the code below:

# Cross Validation
def cross_validation_score(model):
   text_clf = Pipeline([('vect', CountVectorizer()),
                  ('tfidf', TfidfTransformer()),
                  ('clf', model),])
   #text_clf = text_clf.fit(X_train, Y_train)
   #Y_pred = text_clf.predict(X_test)
  return cross_validation.cross_val_score(text_clf, X_test, Y_pred, cv=7)

print('\n################Cross Validation Score###########\n')
for i in models:
  scores=cross_validation_score(models[i])
  print(i + ':\n' + str(scores))
  print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
  print('#######################################\n')

and this is the result that I got for the three classifier:

K-Nearest Neighbors: [ 87.5 37.5 62.5 71.42857143 80. 60. 40. ] Accuracy: 62.70 (+/- 35.04)

#

Random Forest Classifier: [ 87.5 87.5 87.5 100. 100. 100. 60. ] Accuracy: 88.93 (+/- 26.30)

#

Decision Tree Classifier: [ 87.5 87.5 87.5 100. 100. 100. 100. ] Accuracy: 94.64 (+/- 12.37)

#

Sorry but I am a little confused, does it mean that I got an accuracy of 95% with the Decision Tree Classifier for a supervised classification. What did change exactly?

Thanks

$\endgroup$
  • $\begingroup$ seems like it, yes. but why not cross validation? $\endgroup$ – jeff Jul 2 '16 at 12:30
  • $\begingroup$ I think this paper addresses the full scope of issues implicit in your question Do we Need Hundreds of Classifiers to Solve Real World Classification Problems? by Fernandez-Delgado, Cernadas and Barro, available here ... jmlr.org/papers/volume15/delgado14a/delgado14a.pdf $\endgroup$ – Mike Hunter Jul 2 '16 at 12:39
  • $\begingroup$ Do I understand correctly that you used different training/test samples in your trials (which would be valid), and got de facto equal results for many of those? If yes I'd be careful about those numbers: there should be a very small chance of getting de facto equal results, except e.g. many of your samples being exactly equal, so there might be something wrong underneath. $\endgroup$ – geekoverdose Jul 8 '16 at 15:48
1
$\begingroup$

Your table confuses me, but if the decision-tree classifier indeed got higher accuracy (73.91%) for the test set, when fit on the training set, than the other two models did, then yes, it's the best-performing classifier. But you don't know with certainty (certitude) that this classifier will do best in the future. Your training set and test set are finite, so the accuracy scores (and $F_1$ scores, and all the other performance measures) are only estimates.

$\endgroup$
1
$\begingroup$

To add to @Kodiologist's answer, as a first step calculate confidence intervals for your results. For the proportions (accuracy, precision, recall) that can be done by using binomial confidence intervals.

As an example, 31 correct of 46 test cases (67 %) has a confidence interval ranging from ca. 53 - 80 % accuracy. This means (without even going into proper tests for difference of proportion) that your observed accuracies are not significantly different based on only 46 test cases. The same is true for precision and recall - however there the additional caveat applies that you can tune false positive vs. false negative rate for each of the classifiers, so you always need to take both into account. (For the F-values I'd do error propagation from precision and recall).

Comparing classifiers based on these proportions of tested cases in general needs huge numbers of test cases to be statistically meaningful: the variance uncertainty due to evaluating only a small number of test cases is very high. Proper scoring rules are much better behaved in their variance properties (and in that they actually have an optimum) than the proportions.

$\endgroup$
0
$\begingroup$

I think something is wrong in your crossvalidation procedure, the difference between decision tree and random forest cannot be so high, as random forest is based on an ensemble of decision trees. And the results for the decision tree are too stable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.