Suppose in the task of classification, I want to select the best classifier. How to do this? My idea is to select the classifier which gives the highest classification accuracy using the test data during testing phase. Is this the correct approach? I found a related question asked Which is the best classifier and with what performance measures? . According to the answer, I should make the decision based on the F1-score on the test data. Why not use the classification accuracy score instead?
If your dataset is imbalanced, a simple accuracy won't be indicative of the performance of your model.
Imagine you have a highly imbalanced binary classification problem (classes are 0 and 1). Suppose you have 1900 examples for class 0 and just 100 examples for class 1. If you created a dummy classifier that just predicted the class 0, you would achieve a 95% accuracy. In order to solve this problem you should choose a metric that is more insensitive to class imbalance (F1-score is such a metric).
If your dataset is fairly balanced, accuracy should work just fine. Other than that your approach is correct.
No Free Lunch Theorem (Wolpert-MacReady). In the universe of all cost functions, there is no one best classfier.
Ugly Duckling Theorem (Watanabe). In the unviverse of all feature sets, there is no one best set of features.
Occam's Razor. Simpler is better.
You need to use classifier fusion to construct a group of classifiers and then use majority committee vote. Some classifiers will be strong with certain objects, and break down with other objects. Using other classifiers will allow you fill in when classifiers break down. A more important problem, once you develop an ensemble, is that there is no one best set of classifiers, which is called diversity (see Kuncheva et al).