4
$\begingroup$

I have a binary classification task for German webpages for which I only have positive examples. That is why I use learning from positive and unlabeled examples as described on this page, also known as partially supervised learning.

At the moment, I'm just excluding very rare features which occur only once and very frequent features, i.e., stop words and those which occur in more than 50% of all positive examples.

As for classifiers, I want to test Naive Bayes (the positive example webpages are quite short, so I favor the multivariate version with boolean features) and Support Vector Machines (SVM). I've read that feature selection is not so important for SVM as it does not effect the classification results very much. Is that true?

As I'm not so familiar with feature selection algorithms, can you recommend an algorithm that works especially well with features from positive examples only and which generally yields better results than just cutting off very rare and very frequent features? If it is not possible to give a general answer to this question and if it highly depends on my data set, then please say so as well. Thanks a lot!

$\endgroup$
3
$\begingroup$

My experience with SVM has shown it to be fairly robust to uninformative features. Other classifiers, like Naïve Bayes, for example, tend to be more sensitive to such features, making feature selection a relatively more important part of the classification workflow. In terms of feature selection algorithms, I'm a fan of using information theory-based metrics, like mutual information. When I use this, I usually generate a graph of the distribution of the input feature's mutual information during cross-validation, and then manually select some cut-off.

$\endgroup$
  • 1
    $\begingroup$ Please correct me if I'm wrong, but for calculating mutual information you need data from two classes, right? I only have data from the positive class. How to do efficient feature selection having only this one class? Or do I have to apply the partially supervised learning algorithms first to get a set of reliable negative data and then perform mutual information on both this negative and the positive data? $\endgroup$ – pemistahl Oct 10 '12 at 19:28
  • $\begingroup$ You're right, you need two classes for mutual information. For a similar problem, I used active learning techniques to identify my negative-class data. Your partially supervised approach should work as well. $\endgroup$ – Kyle. Oct 10 '12 at 19:36
  • $\begingroup$ Okay, so doing feature selection except for cutting off rare and frequent terms before identifying the negative data does not make sense. That was my main question (apparently, I didn't make it clear enough). Thank you very much! $\endgroup$ – pemistahl Oct 10 '12 at 20:19
  • 1
    $\begingroup$ +1 The SVM is an approximate implementation of a generalisation bound that is independent of the dimensionality of the feature space, so there is good reason to suppose it will work well with redundant features. Selecting the features using e.g. cross-validation invalidates that bound, and is very likely to result in over-fitting the CV error, so quite often feature selection will make the model worse rather than better (although of course the CV error that you have optimised will not reflect that). $\endgroup$ – Dikran Marsupial Jan 8 '13 at 15:39
1
$\begingroup$

In order to make feature selection with unlabeled data, what about clustering+ANOVA to study which variables are more important? With the clustering you learn the distribution from the data, and using ANOVA will let you know which variables are more interesting for your study.

In addition, you may use ANOVA on all your data to see which variables are able to explain your variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.