2
$\begingroup$

These are the results of a classification problem using decision tree, naive bayes and 1-nearest-neighbor as classifiers. There are 10,000 data objects and these results were validated using 10-fold cross validation.

+---------------------+--------------------------------------+--------------------+
|     classifier      | mean share of correct classification | standard deviation |
+---------------------+--------------------------------------+--------------------+
| decision tree       | 85.5 %                               | 5.1 %              |
| naive bayes         | 83.9 %                               | 5.3 %              |
| 1-nearest-neighbor  | 70.1 %                               | 4.9 %              |
+---------------------+--------------------------------------+--------------------+

I must admit I am not competent enough to interpret any results so please help me understand. I have several questions:

  1. Do these results imply that a decision tree is better suited for the classification task than the 1-nearest-neighbor? Why is that? Do I just look at correct classification? What does the standard deviation tell me?
  2. Would I come to the same conclusions if the data set would only contain 50 objects? Please explain why or why not.
$\endgroup$
  • 1
    $\begingroup$ Further relevant information for interpretation could be: Number of classes and distribution of classes in the training data. $\endgroup$ – Nikolas Rieble Feb 9 '17 at 9:16
  • $\begingroup$ @Nikolas Rieble Do you recommend any sources that explains this relationship of classes and distribution? $\endgroup$ – EliteRaceElephant Feb 10 '17 at 23:00
  • 1
    $\begingroup$ There is no specific sources, yet I want to emphasize that accuracy as in (mean share of correct classification) might be overly optimistic in case of highly imbalanced data. See for questions in this forum dealing with imbalance and other evaluation criteria for classification (AUC, recall, precision, etc.) $\endgroup$ – Nikolas Rieble Feb 11 '17 at 15:33
2
+50
$\begingroup$

Do these results imply that a decision tree is better suited for the classification task than the 1-nearest-neighbor? Why is that? Do I just look at correct classification? What does the standard deviation tell me?

The mean share of correct classifications is the average accuracy of the model. The average accuracy of the tree is 85%, much higher than the 1-nn 70%. So, if accuracy is the performance measure by which you select, you should indeed prefer the tree.

The standard deviation measure how stable are the results. Suppose that you had two models with the same accuracy but one with high deviation and one with a low one. The one with high deviation has higher probability to have performance far for the mean (for better and worse). If you are risk adverse you should prefer the model with lower standard deviation.

In your case the standard deviation is almost the same for all models. The accuracy of the tree is about (85 -70)/5 = 3 standard deviation higher than the one of the 1-nn so the gap is quite robust.

Would I come to the same conclusions if the data set would only contain 50 objects? Please explain why or why not.

The smaller your data set, the less certain are the result. Suppose that you have 50 samples and 500 models, it is likely that bad model will be able to give good results. The standard deviation formula takes into account the samples size so in this case you are covered.

You should note that accuracy isn't always the most suitable method to choose a classifier. If you are diagnosing cancer, you probably prefer a classifier with high recall. If you are predicting crime, you might prefer higher precision.

It is also important to consider the model complexity. The lower the model complexity, the more reliable the results. k-nn has high complexity, NB has a low (and quite fixed) complexity and tree have a complexity that highly depends on the data but can be controlled to be not too high.

There are other properties of the models that might be important in some uses cases. For example, trees, especially small, are easy to understand while k-nn are not.

Looking in you case, trees are leading with respect to accuracy, complexity and interpretability so they seem to be the best choice.

$\endgroup$
  • $\begingroup$ Thank you for answering. Would I come to the same conclusion if I compare naive bayes and 1-nearest-neighbor? Naive bayes has the highest standard deviation compared to the others but I would claim it is within acceptable limits since the deviations are almost the same. $\endgroup$ – EliteRaceElephant Feb 10 '17 at 23:04
  • 1
    $\begingroup$ NB is about 2 std from 1-nn so its accuracy is indeed preferable. With respect to model size, it is usually very low. With respect to interpretability, NB it is also better than 1-nn, though NB shows clearly what are the important features, and doesn't show (or uses) relations between multiple feature. $\endgroup$ – DaL Feb 12 '17 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.