5
$\begingroup$

I have a binary classification task for German webpages for which I only have positive examples. That is why I use learning from positive and unlabeled examples as described on this page, also known as partially supervised learning.

At the moment, I'm just excluding very rare features which occur only once and very frequent features, i.e., stop words and those which occur in more than 50% of all positive examples.

As for classifiers, I want to test Naive Bayes (the positive example webpages are quite short, so I favor the multivariate version with boolean features) and Support Vector Machines (SVM). I've read that feature selection is not so important for SVM as it does not effect the classification results very much. Is that true?

As I'm not so familiar with feature selection algorithms, can you recommend an algorithm that works especially well with features from positive examples only and which generally yields better results than just cutting off very rare and very frequent features? If it is not possible to give a general answer to this question and if it highly depends on my data set, then please say so as well. Thanks a lot!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

My experience with SVM has shown it to be fairly robust to uninformative features. Other classifiers, like Naïve Bayes, for example, tend to be more sensitive to such features, making feature selection a relatively more important part of the classification workflow. In terms of feature selection algorithms, I'm a fan of using information theory-based metrics, like mutual information. When I use this, I usually generate a graph of the distribution of the input feature's mutual information during cross-validation, and then manually select some cut-off.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Please correct me if I'm wrong, but for calculating mutual information you need data from two classes, right? I only have data from the positive class. How to do efficient feature selection having only this one class? Or do I have to apply the partially supervised learning algorithms first to get a set of reliable negative data and then perform mutual information on both this negative and the positive data? $\endgroup$
    – pemistahl
    Commented Oct 10, 2012 at 19:28
  • $\begingroup$ You're right, you need two classes for mutual information. For a similar problem, I used active learning techniques to identify my negative-class data. Your partially supervised approach should work as well. $\endgroup$
    – Kyle.
    Commented Oct 10, 2012 at 19:36
  • $\begingroup$ Okay, so doing feature selection except for cutting off rare and frequent terms before identifying the negative data does not make sense. That was my main question (apparently, I didn't make it clear enough). Thank you very much! $\endgroup$
    – pemistahl
    Commented Oct 10, 2012 at 20:19
  • 1
    $\begingroup$ +1 The SVM is an approximate implementation of a generalisation bound that is independent of the dimensionality of the feature space, so there is good reason to suppose it will work well with redundant features. Selecting the features using e.g. cross-validation invalidates that bound, and is very likely to result in over-fitting the CV error, so quite often feature selection will make the model worse rather than better (although of course the CV error that you have optimised will not reflect that). $\endgroup$
    – Dikran Marsupial
    Commented Jan 8, 2013 at 15:39
1
$\begingroup$

In order to make feature selection with unlabeled data, what about clustering+ANOVA to study which variables are more important? With the clustering you learn the distribution from the data, and using ANOVA will let you know which variables are more interesting for your study.

In addition, you may use ANOVA on all your data to see which variables are able to explain your variance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.