Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 7 down vote favorite

I have a data set of a bag of words. I randomly choose some points and use them for testing and the others are used for training.

  • case (1) I just take each data-point from the test set and classify it as having the same class label as its nearest point from the train set.
  • case (2) I do the classification using any known supervised classifier.

I always get better recognition rate in case (1). That is, not doing any learning at all, is better than using any supervised learning, for this data set (and others) ! Is that a frequent situation ?

share|improve this question
I don't know if it's frequent, but it's happened to me. Your classifier just isn't working well. In particular, I have often done tree models and found that the root node is the best tree (depending on definition of "best"). – Peter Flom Nov 4 '12 at 18:10
1  
It's very common for people to make mistakes when they implement learning algorithms. If you use gradient descent, check that a small step in the direction of the gradient actually decreases the cost function by about the amount it should. This is an easy test, but many people skip it and waste time tweaking an algorithm with a sign error, or some similar problem. – Douglas Zare Nov 4 '12 at 18:22
@DouglasZare I've tested with many classifiers from Weka, and others that I implemented myself. For many datasets, using a classifier with a training step (i.e. case(2)) will give better results than case(1), however, for the three datasets of bag of words that I'm currently testing on, I get better results in case(1). I don't think that learning/classifier is not working well, since I've tested with many classifiers. – shn Nov 4 '12 at 19:19

1 Answer

up vote 10 down vote accepted

It is not true that you are not doing any learning. What you are doing is using the well known classification algorithm called Nearest Neighbor (NN). It is important to realize that you are learning as long as you are using the train data (even if you dont explicitly calculate some parameter) - and in this case you are definitely using it.

It is ok that NN is doing well. However, in some cases it may be a sign that there is a problem with your data. This can happen when your data is not IID. For example, in some cases you may have exact or close duplicates in your data. In such a case, many instances in the test set will have a close neighbor in the train set and you will get a high success rate but in fact you are overfitting, because if you get a new point without duplicates your performance will be worse. What you can do in this case is try to remove duplicates in advance, or construct the train/test sets such that duplicates (or tight clusters) have to be in the same set. It is important to look at the data and try to understand what is going on.

share|improve this answer
I'm directly using the training set to do the classification of the test set points. There is no training phase that was performed on the training set. I didn't learnt nothing, I just classified my test points. I don't know why you call this "learning" just because the train set is used. Nonetheless, I just checked the dataset and you are right, there are some duplicated data-points, sometimes the same data-point is in the train and test set, this is not the case for all data-points, but I'll try to fix remove the duplicates and see if the problem is fixed. – shn Nov 4 '12 at 19:46
1  
@shn it is a common mistake to think that you are not learning and that there are no parameters in such a method. As long as you use the training data, it is learning. What you are actually doing is using the whole training set as your "learned parameters", so when you are saving it for later use you are actually "training" (this is why NN is often more prone to overfitting - it actually has a lot of "parameters"). If the predictions you make are dependent on the training set, it is learning. A case with no training would be if you would make predictions WITHOUT using the training set at all. – Bitwise Nov 4 '12 at 19:50
Ok, the problem came from the duplicated points. By removing them, some classifiers achieve a slightly better recognition rate than the NN strategy. However, I didn't noticed that there was too much duplicated points, I removed them and I end-up with a much smaller dataset, the number of instances is not really sufficient to perform an online learning. Do you know any available labelled dataset about document classification which is ready to use (i.e. that I can use without doing preprocessing and much stuff ...) ? There is a great bag-of-words dataset on UCI repo, but labels are not provided. – shn Nov 4 '12 at 22:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.