I have a imbalanced dataset in a binary classification task, where the positives amount vs negatives amount is $0.3\%$ vs $99.7\%$. The gap between positives and negatives is huge. When I train a CNN with the structure used in MNIST problem, the testing result shows a high False Negative Rate. Also, the training error curve goes down quickly in a couple of epochs at the beginning but remains the same value in the following epochs.
How do I process this problem?
This happens because when you take a mini-batch, it is very very less likely (given the ratio of the proportions here) that a mini batch will contain samples of your positives at all. So it will end up learning the pattern for the negative class and after a couple of epochs, everything just gets classified as negative class.
There are two possible ways to handle such a situation.
Given the proportions as 0.3% to 99.7%, this is a very highly skewed data set. You hardly have 3 samples of positive classes for every 1000 samples. I would say you should look at balancing the data set by getting more positive classes. Go and get as many positive samples as you can. Then, you can use a more balanced dataset. For example, you could get 1000 positive samples and then pick a random set of 1000 negative samples and build the classifier. Now, it should be able to learn both the classes.
Use a weighted error measure when updating the weights after a mini-batch. The weights are updated in proportions to the number of samples of the positive and negative classes during any mini-batch. Now, in the current situation given the proportions as 3:1000, even this trick may not work. So you may try getting the proportions to something like 300:1000 by getting 297 more positive samples and combining them with 1000 negative samples. Then with 300:1000 ratio you should weight the error during mini-batches based on number of samples in each of the classes. This should work.
Why do you want to use CNNs here? Have you considered other models that actually handle imbalanced data?
For example, I've found the following two techniques have worked really well for me:
Random Forests w/ SMOTE Boosting: Use a hybrid SMOTE that undersamples the majority class and over-samples the minority class by adjustable percentages. Select these percentages depending on the distribution of your response variable in the training set. Feed this data to your RF model. Always cross-validate/perform grid-search to find the best parameter settings for your RFs.
XGBoost w/ hyper-parameter optimisation: Again, cross-validate or perform gird-search to find the best parameter settings for the model. Additionally, xgboost allows you to balance positive and negative class weights using scale_pos_weight. See the parameter documentation for a complete list.
I should also add that the data-set I was working on had the same percentage of skew and I was able to obtain Sensitivity score of $0.941$ and a Specificity of $0.71$ with xgboost, which means the model is predicting the true positives quite accurately and that bodes well for me.
Unbalanced dataset is a common issue in all areas and does not specifically concern computer vision and problems dealt by Convolutional Neural Networks (CNNs).
To tackle this problem you should try to balance your dataset, either by over-sampling minority classes or under-sampling majority classes (or both). Arguably, a good choice would be SMOTE (Synthetic Minority Over-sampling Technique) algorithm, as mentioned above. Here you can find a comparison of different over-sampling algorithms. If you're a Python user, imbalanced-learn is a nice library that implements many useful techniques for balancing datasets.
On the other hand, if you're trying to classify images, a nice way to increase your dataset size is to augment it (i.e. by creating reasonable synthetic examples, e.g. similar images but rotated/shifted tiny bit with respect to original ones). You might sometimes find it useful to augment the minority classes to achieve better balance. Keras ImageDataGenerator class is a good tool for this purpose.
