Which classifiers work well with unbalanced data?

Question

I have a binary classification problem which is very unbalanced - it can have 98% of data from one class. Which classifiers work well with this sort of data?

I have an unlimited supply of training data, since I produce it using a pseudo random number generator. However, I found that to get a neural network to produce decent results, I had to generate balanced (50:50) data. This is the equivalent of over-sampling. The problem with this approach is that the training data is then not representative of real life.

Answer 1

Some options:

• Do not use accuracy alone as a metric. That way, we would get 98% accuracy with everything classified as the majority class, which would not mean anything. Precision & Recall might be a better one.
• You could try using a Cost sensitive classifier through which you can state the cost of misclassification of the different classes.
• Use an SVM but penalize one of the classes which can be done using LibSVM
• boost the number of minority class training examples by artificially creating new samples from the existing samples.
• resample the set, to have a proportional number of samples in both the classes (probably not an option in your case)

Answer 2

Just to add to @tdc's reply: the problem with skewed data is that models tend to over-fit and result in biased models. For some of them there are modifications to the original formulation that help correcting that bias. For example, there are alternative formulations of naive Bayes, kNN which deal with this problem.

As he points out: the problem is that usually one attempts to maximize the accuracy. And that makes the classifier "focus" on the most populous classes, since that yields better accuracy.