Binary classifications problem where number of negative instances is much larger than number of positive instances I am trying to write a logistic classifier that will classify text tokens on a web page. One problem I am running into is the fact that their are many more negative instances of the tokens I am trying to classify then positive ones. My question is: what is the ratio of positive training data to negative training I should be using? It seems that the more neg
I tried googling the problem to look for relevant papers, but had no success. 
 A: I ran into a similar problem a year ago and wanted to know if imbalanced class labels lead to problems even in the perfect case of training and test data coming from the same distribution, so I ran a little experiment:
I trained a classifier (boosted classification trees) on a 6-dimensional simulated data set (50,000 rows total, 75% for training, 25% for testing) with varying signal-to-noise-ratio (4 values for the SNR), varying class label imbalance (10 values) and 3 "sampling" schemes, i.e. ways to deal with the imbalance: 1.) ignore it ("none"), 2.) random undersampling ("rus") leading to a balanced data set, typically much smaller than the original, and 3.) attaching weights to the observations inversely proportional to their frequency.
I have graphed these 4 sets of 30 (10x3) test errors as measured by the AUC on this plot: http://dl.dropbox.com/u/8089659/ImbalancedData.pdf
I ran out of patience and did not average the RUS results over multiple runs, which is why it is so noisy, but clearly for minority class fraction less than ~ 0.15-0.2 the undersampling and the weights seem to outperform the normal classifier which is trained on the entire data set.While the effect is not huge, it appears consistent. The beauty of RUS is how easy one can parallelize the different samples.
The R script to create the plots and run the experiments is at:
http://dl.dropbox.com/u/8089659/Imbalance.R
A: I believe you should use the ratio of positive to negative events that reflects your data. Logistic regression does not assume anything about the distribution of the DV, and if you change it, your logistic model will model the wrong thing.  The problem is if the N in the smaller group (for you, the positive responses) is small, especially if there are many potential IVs. But that is not solved by changing the number of negative events.
