I am relatively new to Machine Learning and would appreciate your insights. I need to classify a set of keywords into five categories. I thought about using linear SVM for the exercise. I have a training dataset that contains the keywords and the classification and a second dataset that needs to be classified and only contains the keywords. What I know is that the unconditional distribution of the five categories is different in the training data than it is in the data that needs to be classified. How does that impact my learner?

  • $\begingroup$ Depends by what you mean as distribution. Do you mean the proportion of each of the five categories appears differently than the proportion of the appearance of each category in 'real' data? If yes that probably isn't a big problem, although it depends. If you mean that the relationship between keywords and category is somewhat different in training than 'real' data, then you have a big problem. A good rule of thumb is having your data be distributed as close to possible as the 'real' data. $\endgroup$ – user3494047 Sep 21 '16 at 15:42
  • $\begingroup$ The training data comes from a different DGP. But the rules for assigning observations to categories are the same in the two datasets. However, the proportions of the five categories are quite different. How should I deal with this? $\endgroup$ – Petrvalsky Sep 21 '16 at 15:45
  • $\begingroup$ What does DGP mean? $\endgroup$ – user3494047 Sep 21 '16 at 15:47
  • $\begingroup$ In general if the rules for assigning categories are the same, which what i mean is that for every input set of keywords the probability that this set belongs to each category is the same in the training data as the 'real' data, then it should be fine to train a classifier with this data. As long as you have enough examples of each category, train your SVM classifier and measure your performance and see if it is good enough for you. $\endgroup$ – user3494047 Sep 21 '16 at 15:48
  • $\begingroup$ Data Generating Process $\endgroup$ – Petrvalsky Sep 21 '16 at 15:48

In general a classifier (like a one-vs-all svm classifier) is trying to learn one function for each category. That function is supposed to receive an input of a set of keywords, as you describe it, and output a score, which is correlated to how likely the input set of keywords belongs to a specific category.

Every example that you give the classifier that is part of the category, lets the classifier know what kind of input belongs to the category. Every example which is labeled as not part of the category lets the classifier know what kind if input doesn't belong. The more different kinds of data that you give the classifier the better.

Theoretically, if you have no noise in your data and you give it all possible inputs, then your classifier should know how to classify perfectly. So a general rule of thumb is that the more of different kinds of data you give it the better. On the other hand, you classifier can't do everything and you don't want to overfit. if you're using a linear svm then your classifier can only draw straight lines in the input space and make decisions based on those lines. If 95% of the 'real' data that is part of category 1 is in region A and 5% of the real data that is in category 1 is in region B, and assume due to the choice of classifier, your classifier can only classify one of the regions as category 1, then you want it to find region A and ignore region B. You will have to live with 5% of category 1 data to be misclassified.

So this is where the proportion of data does matter. The more examples from region A you give it, the more likely it is to choose region A as its classify as category 1 region. The more examples from region B you give it, the more likely it is to choose region B as the area that if your input is there, it will classify it as category 1.

So what I recommend you do, is keep this in mind, do some data exploration to see how the data behaves (answering how to do this is a different question), choose a method (you said you were going to use SVM, I assume you have good reason to do so), test your performance on real data and see if it is acceptable. I'm not able to say exactly what you should do because it depends on the data, and from the behavior of the data you should choose what data preprocessing techniques and classifiers to use.

Sorry if my explanation of regions is confusing. I'm not good at drawing things, although it probably would have helped my explanation, so if someone can help or do a better job, please do.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.