Background: Previously, I ran my elastic net model on class-imbalanced data. I found out this is bad practice generally, so I downsampled the data to resolve the class imbalance.
probsTrain = predict(blocks_elastic_net_model_DS, blocks_training, type = "prob") rocCurve = roc(response = blocks_training$trophout1, predictor = probsTrain[, "Trophy"], levels = rev(levels(blocks_training$trophout1))) plot(rocCurve, print.thres = "best") rocCurve [ROC CURVE PLOT] probsTest <- predict(blocks_elastic_net_model_DS, blocks_testing, type = "prob") threshold <- 0.595 # TODO AA: This value was chosen from the above ROC graph? pred <- factor(ifelse(probsTest[, "Trophy"] > threshold, "Trophy", "NoTrophy")) confusionMatrix(pred, blocks_testing$trophout1, positive="Trophy", mode="everything")
My model no longer has the class imbalance issue (since I downsampled the dataset to train it), but my original training and test datasets are still imbalanced. So, whenever I generate the confusion matrix it really shows the how my model---trained on class-balanced data---performs on class-imbalanced data. I can't decide if there's an issue with this. Part of me thinks that it shouldn't cause issues because the model is still trained on features meant help it differentiate between Trophy and NoTrophy outcomes (my binary outcome measure), and so regardless of class balance those differentiating features should be similar for Trophy vs. NoTrophy...right? However, if the model is trained on class-balanced data (downsampled), then isn't it "expecting" to be tested on class-balanced data? And so might it be inappropriate to test that model on class-imbalanced data?
TLDR: I downsampled my data before training the model on it to get rid of class imbalance. Now, whenever I test the model on my data (which in reality is class-imbalanced, hence why I had to downsample it), is my model still valid?
- Why does the data need to be class-balanced before I train the model on it? Can't it be trained on class-imbalanced data and still learn to make good predictions?
- Is there any theoretical issue with testing a model on class-imbalanced data when that model was trained on class-balanced data (or vice versa)?