Suppose I want to learn a classifier that predicts if an email is spam. And suppose only 1% of emails are spam.
The easiest thing to do would be to learn the trivial classifier that says none of the emails are spam. This classifier would give us 99% accuracy, but it wouldn't learn anything interesting, and would have a 100% rate of false negatives.
To solve this problem, people have told me to "downsample", or learn on a subset of the data where 50% of the examples are spam and 50% are not spam.
But I'm worried about this approach, since once we build this classifier and start using it on a real corpus of emails (as opposed to a 50/50 test set), it may predict that a lot of emails are spam when they're really not. Just because it's used to seeing much more spam than there actually is in the dataset.
So how do we fix this problem?
("Upsampling," or repeating the positive training examples multiple times so 50% of the data is positive training examples, seems to suffer from similar problems.)