I'm developing a predictive model on a dataset of about 25K observations with the response variable having ~60 classes. There are ~130 predictor variables, all of them binary. In this problem I care about predicting the right probabilities on the response variable and not about predicting one single value.
I'm dealing with an imbalanced dataset where the least common class in the response variable has a relative frequency of 0.004% and the most common one has 12%.
I've been trying different models using up-sampling inside the cross-validation training sets and have achieved a ROC of 80% on the unbalanced hold-out sets using Random Forests which is pleasantly good.
However, wouldn't up-sampling introduce error to the predicted probabilities? I'm thinking so because the probabilities are calculated with the training set which is artificially balanced. Another approach I'm considering is assigning higher weights to the uncommon cases when training the model. But wouldn't that be effectively the same as up-sampling?