2
$\begingroup$

Taking a long shot here. So I have a a small dataset of ~500 images with discrete labels from 1 to 9. My task is to detect the per-class and overall accuracy of this classification method using a (deep) neural network.

I get an accuracy of 56% using multinomial logistic regression using train/test split method. I used transfer learning in PyTorch for some of the categories that have >25 images (2, 3, 5, 7, and 9). However, I do not know how to use this tutorial and report test accuracy when I only have like 3 or 11 images in a category.

https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

Here is the number of images per category:

category 1: 3
category 2: 66
category 3: 175
category 4: 18
category 5: 54
category 6: 12
category 7: 91
category 8: 11
category 9: 38

What (deep) neural network method (hopefully with a link to tutorial, would you suggest that could work with a dataset like mine that has severe data imbalance and also some categories are having <10 images?

I have provided further discussion of the problem I am dealing with as well here: https://discuss.pytorch.org/t/leave-one-out-cross-validation-for-images-with-labels-and-discrete-confidence-levels-for-labels/29227 Here is the confusion matrix I get from multinomial logistic regression: enter image description here

$\endgroup$
6
  • 3
    $\begingroup$ Have you considered the option of "blowing up" your training set by applying transforms to some of the images to create a bigger training set? $\endgroup$
    – mroman
    Nov 11, 2018 at 19:34
  • 2
    $\begingroup$ Data augmentation seems like a good idea indeed (+1 to mroman). Comment: with 3 and 11 instances (as in categories 1 and 8) most "within-class" accuracy metrics are you know... a bit irrelevant? What will be it be their standard error like? 50%? $\endgroup$
    – usεr11852
    Nov 11, 2018 at 23:40
  • $\begingroup$ @usεr11852 can you please explain what you mean by 'a bit irrelevant? What will be it be their standard error like? 50%?' $\endgroup$
    – Mona Jalal
    Nov 12, 2018 at 0:06
  • $\begingroup$ @mroman I have used 'data augmentation' in the past such as rotation/adding noise/etc as given by DL frameworks. However, just having 3 or 11 sample images to start with is really low, isn't it? $\endgroup$
    – Mona Jalal
    Nov 12, 2018 at 0:06
  • $\begingroup$ @usεr11852 also, you say within-class accuracy is a bit irrelevant. Do you mean "per-class accuracy" such as reported in confusion matrix is irrelevant? what accuracy metric do you suggest? $\endgroup$
    – Mona Jalal
    Nov 12, 2018 at 0:08

2 Answers 2

1
$\begingroup$

There are several methods to counter imbalanced datasets. Yet, with so few samples in several of the classes I'm not sure if you will ever get a satisfactory classification. I'm not sure if you can learn class features from only 3 samples as in class 1. In any case, this might lead to extreme overfitting or at least not a good generalization.

Nevertheless, you can try the following methods:

  • Upsampling, i.e. simply doubling items from the minority classes

  • Data augmentation (as mentioned in the comments), i.e. rotating, scaling, cropping, flipping,... the images (you have to decide which modifications make sense with your application)

  • Penalizing / cost sensitive learning / class weights, i.e. penalizing the miss-classification of a minority class item more severely than that of a majority class

Disclaimer: I'm not that familiar with Pytorch as I myself work with Keras and Tensorflow most of the time. Maybe someone else can provide some Pytorch specific examples or tutorials. In any way, the keywords should at least guide you into the right direction.

$\endgroup$
1
$\begingroup$

The imbalance in your problem almost certainly is not a problem [1, 2, 3, 4]. However, the sheer lack of samples is. I see two paths forward.

  1. Combine rare categories into other categories that are "similar enough" (however you define that). Frank Harrell discusses this in his Regression Modeling Strategies textbook.

  2. Use data augmentation techniques like slight rotations. The trouble here is that you then distort the class distribution (the "prior" probability of each category), so a weighted loss function might be in order to counteract the over-abundance of members of these rare categories.

Note that if you have binned numerical values into nine categories, there are possibly severe problems with having done so instead of treating the problem as a regression. Nothing indicates that you have done so, but people do it, so this is worth a mention.

$\endgroup$
2
  • $\begingroup$ Somewhat tangential, you overall have a real lack of data for doing deep learning. A mere $500$ images seems very small. Can you acquire a billion more images? $\endgroup$
    – Dave
    Apr 18 at 16:35
  • 3
    $\begingroup$ I disagree that imbalance is almost certainly not a problem. If you have very small datasets (as measured by the size of the smallest class) then the variance in estimating the model parameters can lead to an undue bias against the small classes. This seems to be such a case, especially using a large model (with many parameters to estimate) and a large attribute space. Gathering more data may be the only solution for that. I would use a pre-trained model (trained on billions of other images) and then retrain the output layer as if it were an MLR model. $\endgroup$ Apr 18 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.