Class Imbalence Problem even after Balancing Data

Question

So I am training a neural network on a binary classification problem and my Case (1) and Controls (0) were imbalanced so I oversampled my cases so that that the training set was 0.5053 made up of controls. I did not balance my test set which was 0.562 controls. In the beginning my train and test accuracy rises (it is not very accurate still but I expect this to be the case) but then the training accuracy steeply drops while the test accuracy plateaus.

They end up with accuracies of 0.5053 and 0.562 respectively so the network is just classifying everything the same. I do not understand how this behavior arises as I thought that balancing my training set would avoid the problem of classifying everything the same? Also, the training set begins to learn initially upwards from 50/50 but I cannot understand its reversion. Is there anything I can do to prevent this? Or should I just employ early stopping when the training accuracy begins to decrease?

Any insight would be appreciated!

Answer 1

It sounds like you are using a neural network. As an aside, apart from over-sampling, you could use a SMOTE technique, and generate synthetic data. You results are curious, given that you have already balanced your data. I don't have a direct answer.

First thing I would check is that you really are balancing them correctly. I would also review whether you are using the most appropriate loss function. Try alternating between different loss functions to see whether this issue manifests equally between them. However, accuracy probably isn't the best performance metric to use for imbalanced data like this. Instead I would look into using both accuracy and recall, and you'll probably end up on f1 score as a result. Also, consider AUC too (I just noticed that you are already looking at this).

I often find gradient boosted trees do better on heavily imbalanced problems where I don't want to perform any major data sampling. In part, this is because you have a direct parameter which informs the model a priori how the classes are imbalanced. For example, XGB uses parameter Scale_pos_weight. This is the ratio of number of negative class to the positive class.