Classification error is lower when I don't do any learning on the dataset? I have a data set of a bag of words. I randomly choose some points and use them for testing and the others are used for training.


*

*case (1) I just take each data-point from the test set and classify it as
having the same class label as its nearest point from the train set.

*case (2) I do the classification using any known supervised classifier.


I always get better recognition rate in case (1). That is, not doing any learning at all, is better than using any supervised learning, for this data set (and others) ! Is that a frequent situation ?
 A: It is not true that you are not doing any learning. What you are doing is using the well known classification algorithm called Nearest Neighbor (NN). It is important to realize that you are learning as long as you are using the train data (even if you dont explicitly calculate some parameter) - and in this case you are definitely using it.
It is ok that NN is doing well. However, in some cases it may be a sign that there is a problem with your data. This can happen when your data is not IID. For example, in some cases you may have exact or close duplicates in your data. In such a case, many instances in the test set will have a close neighbor in the train set and you will get a high success rate but in fact you are overfitting, because if you get a new point without duplicates your performance will be worse. What you can do in this case is try to remove duplicates in advance, or construct the train/test sets such that duplicates (or tight clusters) have to be in the same set. It is important to look at the data and try to understand what is going on.
