using (deep) neural networks for a severely imbalanced image dataset when some classes have <10 images

Question

Taking a long shot here. So I have a a small dataset of ~500 images with discrete labels from 1 to 9. My task is to detect the per-class and overall accuracy of this classification method using a (deep) neural network.

I get an accuracy of 56% using multinomial logistic regression using train/test split method. I used transfer learning in PyTorch for some of the categories that have >25 images (2, 3, 5, 7, and 9). However, I do not know how to use this tutorial and report test accuracy when I only have like 3 or 11 images in a category.

https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

Here is the number of images per category:

category 1: 3
category 2: 66
category 3: 175
category 4: 18
category 5: 54
category 6: 12
category 7: 91
category 8: 11
category 9: 38

What (deep) neural network method (hopefully with a link to tutorial, would you suggest that could work with a dataset like mine that has severe data imbalance and also some categories are having <10 images?

I have provided further discussion of the problem I am dealing with as well here: https://discuss.pytorch.org/t/leave-one-out-cross-validation-for-images-with-labels-and-discrete-confidence-levels-for-labels/29227 Here is the confusion matrix I get from multinomial logistic regression:

Answer 1

There are several methods to counter imbalanced datasets. Yet, with so few samples in several of the classes I'm not sure if you will ever get a satisfactory classification. I'm not sure if you can learn class features from only 3 samples as in class 1. In any case, this might lead to extreme overfitting or at least not a good generalization.

Nevertheless, you can try the following methods:

• Upsampling, i.e. simply doubling items from the minority classes

• Data augmentation (as mentioned in the comments), i.e. rotating, scaling, cropping, flipping,... the images (you have to decide which modifications make sense with your application)

• Penalizing / cost sensitive learning / class weights, i.e. penalizing the miss-classification of a minority class item more severely than that of a majority class

Disclaimer: I'm not that familiar with Pytorch as I myself work with Keras and Tensorflow most of the time. Maybe someone else can provide some Pytorch specific examples or tutorials. In any way, the keywords should at least guide you into the right direction.

Answer 2

The imbalance in your problem almost certainly is not a problem [1, 2, 3, 4]. However, the sheer lack of samples is. I see two paths forward.

1. Combine rare categories into other categories that are "similar enough" (however you define that). Frank Harrell discusses this in his Regression Modeling Strategies textbook.

2. Use data augmentation techniques like slight rotations. The trouble here is that you then distort the class distribution (the "prior" probability of each category), so a weighted loss function might be in order to counteract the over-abundance of members of these rare categories.

Note that if you have binned numerical values into nine categories, there are possibly severe problems with having done so instead of treating the problem as a regression. Nothing indicates that you have done so, but people do it, so this is worth a mention.