5
$\begingroup$

Suppose in the task of classification, I want to select the best classifier. How to do this? My idea is to select the classifier which gives the highest classification accuracy using the test data during testing phase. Is this the correct approach? I found a related question asked Which is the best classifier and with what performance measures? . According to the answer, I should make the decision based on the F1-score on the test data. Why not use the classification accuracy score instead?

$\endgroup$
8
$\begingroup$

If your dataset is imbalanced, a simple accuracy won't be indicative of the performance of your model.

Imagine you have a highly imbalanced binary classification problem (classes are 0 and 1). Suppose you have 1900 examples for class 0 and just 100 examples for class 1. If you created a dummy classifier that just predicted the class 0, you would achieve a 95% accuracy. In order to solve this problem you should choose a metric that is more insensitive to class imbalance (F1-score is such a metric).

If your dataset is fairly balanced, accuracy should work just fine. Other than that your approach is correct.

$\endgroup$
  • $\begingroup$ Thank you for your answer. Could you please provide some insight on how to validate once a network say neural network is trained in classification task? Do we calculate the error by running the trained network by using the training data again? $\endgroup$ – Ria George Apr 20 '18 at 20:43
  • $\begingroup$ No, you should keep part of the data hidden and evaluate your data on that (we call this the test set). The reason we do that is that ML algorithms suffer from a problem called overfitting, where they achieve a high performance on the training data, but can't generalize well on new data. By keeping the test set hidden you ensure that your evaluation will be an unbiased estimate of your network's performance. $\endgroup$ – Djib2011 Apr 20 '18 at 22:23
1
$\begingroup$

No Free Lunch Theorem (Wolpert-MacReady). In the universe of all cost functions, there is no one best classfier.

Ugly Duckling Theorem (Watanabe). In the unviverse of all feature sets, there is no one best set of features.

Occam's Razor. Simpler is better.

You need to use classifier fusion to construct a group of classifiers and then use majority committee vote. Some classifiers will be strong with certain objects, and break down with other objects. Using other classifiers will allow you fill in when classifiers break down. A more important problem, once you develop an ensemble, is that there is no one best set of classifiers, which is called diversity (see Kuncheva et al).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.