How to deal with biased dataset for both training and testing data? I am currently working on a classification problem with a highly biased dataset. The dataset is biased for both training and testing data. And I am having trouble dealing with the dataset or modifying the model.
For example, I have 30 classes, 70% of which are class A and B.
And I have tried to expand the dataset to make my model more robust. However, it has worse performance on the test dataset since the test dataset is also biased.
I am using a deep learning model with cross-entropy loss and also tried weighted cross entropy loss. I wonder what else I can try to relieve the impact of the bias.
 A: What did you already try to balance your dataset? Often, basic methods like random over-/undersampling are used, but if this does not help, you might want to try advanced sampling methods. Additionally you need to keep in mind that you should just re-balance your dataset for the training and NOT for the evaluation of your models.
A: A few thoughts:
First, even if you ultimately need to evaluate the accuracy of your model, training and testing your model on accuracy is probably not be the best way to proceed. This issue is discussed extensively on this site, with this page being a good place to start. That probably explains why your cross-entropy losses (log losses) agree much better between your test and training sets than do assessments of accuracy. Stick with cross-entropy.
Second, as you seem to have further processing beyond this initial modeling, consider doing that further processing in a way that carries through the predicted class probabilities until the end rather than depending on an early all-or-none assignment of cases to 1 of your 30 classes. That could lead to more reliable final results. 
Third, you haven't said much about the nature of your "deep learning model." You might need to consider a different type of model, or adjusting the learning characteristics of the model (as with the $\ell 2$ penalization you seem to be considering).
Fourth, it's possible that you just don't have enough data of a type that can discriminate among class memberships, particularly for the low-prevalence classes. Even the best attempts at such problems can hit unavoidable barriers.
A: Still working on the problem of imbalanced dataset recently. I tried to expand the dataset through duplication and random replacement, and also add regularizations but these did not improve the overall accuracy. 
There are some papers talking about dealing with imbalanced dataset and the ideas of customer loss functions, such as focal loss, GHM, DR loss, are adopted in my experiments. I have tried some of them and it turned out they helped a little.
