0
$\begingroup$

I'm developing a predictive model on a dataset of about 25K observations with the response variable having ~60 classes. There are ~130 predictor variables, all of them binary. In this problem I care about predicting the right probabilities on the response variable and not about predicting one single value.

I'm dealing with an imbalanced dataset where the least common class in the response variable has a relative frequency of 0.004% and the most common one has 12%.

I've been trying different models using up-sampling inside the cross-validation training sets and have achieved a ROC of 80% on the unbalanced hold-out sets using Random Forests which is pleasantly good.

However, wouldn't up-sampling introduce error to the predicted probabilities? I'm thinking so because the probabilities are calculated with the training set which is artificially balanced. Another approach I'm considering is assigning higher weights to the uncommon cases when training the model. But wouldn't that be effectively the same as up-sampling?

$\endgroup$
0
$\begingroup$

It makes sense to oversample (upsample), but you will need to adjust the probabilities you get.

These two links have examples of how to adjust the probabilities. They are describing the same adjustment, but go about it in a slightly different way.

Here's an example where we've oversampled (or undersampled) so that the original probability of .05 is now .5 in our training set. So the probability we get out of the prediction model (scored probability) needs to be adjusted back. As you can see, the adjustment is not linear. enter image description here

http://www.data-mining-blog.com/tips-and-tutorials/overrepresentation-oversampling/ https://yiminwu.wordpress.com/2013/12/03/how-to-undo-oversampling-explained/

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.