6
$\begingroup$

I have a data set with ~ 7 million rows, of which ~ 100k are positives. I'm looking to shrink the data by keeping all the positives and then randomly sampling several hundred thousand negative examples to round out the data set.

I'm uncertain what the guidelines are though. Are there any rules of thumb for what % needs to be positive / negative?

Any suggestions would be appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ So what are you trying to achieve? What are your analysis goals? And why do you think you need to subsample? What exactly the problems are with the full data set? $\endgroup$
    – StasK
    Commented Sep 4, 2012 at 23:34
  • $\begingroup$ @Stask: I'm trying to predict whether someone will respond to a mailing (binary classification). My main goal is to work with a data set that is < 1 GB instead of one that is 10 GB. $\endgroup$
    – screechOwl
    Commented Sep 4, 2012 at 23:43
  • $\begingroup$ You don't always have to work with the 10Gb set, you could use methods like a decision tree or random forests to generate the classifier and then work with that; the classifier won't require that much information compared to the information needed to train it. $\endgroup$
    – casperOne
    Commented Sep 5, 2012 at 1:49
  • $\begingroup$ @casperOne - I've found that decision trees trained on rare event data will often return only a single root - I think a logistic regression is preferable. $\endgroup$
    – RobertF
    Commented Dec 4, 2014 at 16:04

1 Answer 1

Reset to default
5
$\begingroup$

If you keep all the positives from your data set you may find that you have skewed your results.

A priori the probability of a positive in your data is about 1 in a hundred. If you down sample so you have 100K +ve and 100K -ve the a priori +ve probability is now 1 in 2.

Unless there is a large separation with little overlap between the two classes you will most likely create a strongly biased classifier.

As a first step create a smaller stratified sub sample and see what performance you can achieve with that.

Then you can investigate how your classifier behaves if you increase the percentage of +ves in the training set and use a test set with a much higher percentage of -ves. This should give you some idea of the sensitivity of your methods to class balance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Let's say you start with an 80:20 split and you train on the 80. You can either start by downsampling the negative class, so you have a 50:50 ratio of positive to negative sample, and slowly lessen how much you down sample. Or vice versa, start with a full class imbalance as above say 3:100, 1:0, and upsample. How do you know, when to stop downsampling? / Upsampling? $\endgroup$
    – Chuck
    Commented Mar 17, 2020 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.