0
$\begingroup$

This question already has an answer here:

I'm developing a predictive model on a dataset of about 25K observations with the response variable having ~60 classes. There are ~130 predictor variables, all of them binary. In this problem I care about predicting the right probabilities on the response variable and not about predicting one single value.

I'm dealing with an imbalanced dataset where the least common class in the response variable has a relative frequency of 0.004% and the most common one has 12%.

I've been trying different models using up-sampling inside the cross-validation training sets and have achieved a ROC of 80% on the unbalanced hold-out sets using Random Forests which is pleasantly good.

However, wouldn't up-sampling introduce error to the predicted probabilities? I'm thinking so because the probabilities are calculated with the training set which is artificially balanced. Another approach I'm considering is assigning higher weights to the uncommon cases when training the model. But wouldn't that be effectively the same as up-sampling?

$\endgroup$

marked as duplicate by Michael Chernick, Stephan Kolassa, kjetil b halvorsen, mdewey, Peter Flom Nov 22 '17 at 13:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

0
$\begingroup$

It makes sense to oversample (upsample), but you will need to adjust the probabilities you get.

These two links have examples of how to adjust the probabilities. They are describing the same adjustment, but go about it in a slightly different way.

Here's an example where we've oversampled (or undersampled) so that the original probability of .05 is now .5 in our training set. So the probability we get out of the prediction model (scored probability) needs to be adjusted back. As you can see, the adjustment is not linear. enter image description here

http://www.data-mining-blog.com/tips-and-tutorials/overrepresentation-oversampling/ https://yiminwu.wordpress.com/2013/12/03/how-to-undo-oversampling-explained/

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.