Does it make sense to up-sample imbalanced data if I care about predicting the right probabilities? [duplicate]

Question

I'm developing a predictive model on a dataset of about 25K observations with the response variable having ~60 classes. There are ~130 predictor variables, all of them binary. In this problem I care about predicting the right probabilities on the response variable and not about predicting one single value.

I'm dealing with an imbalanced dataset where the least common class in the response variable has a relative frequency of 0.004% and the most common one has 12%.

I've been trying different models using up-sampling inside the cross-validation training sets and have achieved a ROC of 80% on the unbalanced hold-out sets using Random Forests which is pleasantly good.

However, wouldn't up-sampling introduce error to the predicted probabilities? I'm thinking so because the probabilities are calculated with the training set which is artificially balanced. Another approach I'm considering is assigning higher weights to the uncommon cases when training the model. But wouldn't that be effectively the same as up-sampling?

Answer 1

It makes sense to oversample (upsample), but you will need to adjust the probabilities you get.

These two links have examples of how to adjust the probabilities. They are describing the same adjustment, but go about it in a slightly different way.

Here's an example where we've oversampled (or undersampled) so that the original probability of .05 is now .5 in our training set. So the probability we get out of the prediction model (scored probability) needs to be adjusted back. As you can see, the adjustment is not linear.

http://www.data-mining-blog.com/tips-and-tutorials/overrepresentation-oversampling/ https://yiminwu.wordpress.com/2013/12/03/how-to-undo-oversampling-explained/