3
$\begingroup$

I am trying to write a logistic classifier that will classify text tokens on a web page. One problem I am running into is the fact that their are many more negative instances of the tokens I am trying to classify then positive ones. My question is: what is the ratio of positive training data to negative training I should be using? It seems that the more neg

I tried googling the problem to look for relevant papers, but had no success.

$\endgroup$
1
  • $\begingroup$ I'd try searching using the terms "logistic regression" and either "narrow distribution" or "rare event." $\endgroup$
    – rolando2
    Oct 6, 2011 at 23:28

2 Answers 2

4
$\begingroup$

I ran into a similar problem a year ago and wanted to know if imbalanced class labels lead to problems even in the perfect case of training and test data coming from the same distribution, so I ran a little experiment: I trained a classifier (boosted classification trees) on a 6-dimensional simulated data set (50,000 rows total, 75% for training, 25% for testing) with varying signal-to-noise-ratio (4 values for the SNR), varying class label imbalance (10 values) and 3 "sampling" schemes, i.e. ways to deal with the imbalance: 1.) ignore it ("none"), 2.) random undersampling ("rus") leading to a balanced data set, typically much smaller than the original, and 3.) attaching weights to the observations inversely proportional to their frequency. I have graphed these 4 sets of 30 (10x3) test errors as measured by the AUC on this plot: http://dl.dropbox.com/u/8089659/ImbalancedData.pdf

I ran out of patience and did not average the RUS results over multiple runs, which is why it is so noisy, but clearly for minority class fraction less than ~ 0.15-0.2 the undersampling and the weights seem to outperform the normal classifier which is trained on the entire data set.While the effect is not huge, it appears consistent. The beauty of RUS is how easy one can parallelize the different samples.

The R script to create the plots and run the experiments is at: http://dl.dropbox.com/u/8089659/Imbalance.R

$\endgroup$
1
  • $\begingroup$ Usually these experiments are carried out with accuracy measure, that is somewhat misleading. Nice choice the AUC. Of course you performed RUS only on the training set, is it true? Do you think that RUS or weighting could improve an F-measure for the minority class too? Thanks $\endgroup$
    – Simone
    Dec 8, 2011 at 17:03
3
$\begingroup$

I believe you should use the ratio of positive to negative events that reflects your data. Logistic regression does not assume anything about the distribution of the DV, and if you change it, your logistic model will model the wrong thing. The problem is if the N in the smaller group (for you, the positive responses) is small, especially if there are many potential IVs. But that is not solved by changing the number of negative events.

$\endgroup$
1
  • $\begingroup$ Sorry I forgot to mention that I am using regularized logistic regression which should be like adding a prior. $\endgroup$
    – user6688
    Oct 6, 2011 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.