0
$\begingroup$

Taking a long shot here. So I have a a small dataset of ~500 images with discrete labels from 1 to 9. My task is to detect the per-class and overall accuracy of this classification method using a (deep) neural network.

I get an accuracy of 56% using multinomial logistic regression using train/test split method. I used transfer learning in PyTorch for some of the categories that have >25 images (2, 3, 5, 7, and 9). However, I do not know how to use this tutorial and report test accuracy when I only have like 3 or 11 images in a category.

https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

Here is the number of images per category:

category 1: 3
category 2: 66
category 3: 175
category 4: 18
category 5: 54
category 6: 12
category 7: 91
category 8: 11
category 9: 38

What (deep) neural network method (hopefully with a link to tutorial, would you suggest that could work with a dataset like mine that has severe data imbalance and also some categories are having <10 images?

I have provided further discussion of the problem I am dealing with as well here: https://discuss.pytorch.org/t/leave-one-out-cross-validation-for-images-with-labels-and-discrete-confidence-levels-for-labels/29227 Here is the confusion matrix I get from multinomial logistic regression: enter image description here

$\endgroup$
  • 3
    $\begingroup$ Have you considered the option of "blowing up" your training set by applying transforms to some of the images to create a bigger training set? $\endgroup$ – mroman Nov 11 '18 at 19:34
  • 1
    $\begingroup$ Data augmentation seems like a good idea indeed (+1 to mroman). Comment: with 3 and 11 instances (as in categories 1 and 8) most "within-class" accuracy metrics are you know... a bit irrelevant? What will be it be their standard error like? 50%? $\endgroup$ – usεr11852 says Reinstate Monic Nov 11 '18 at 23:40
  • $\begingroup$ @usεr11852 can you please explain what you mean by 'a bit irrelevant? What will be it be their standard error like? 50%?' $\endgroup$ – Mona Jalal Nov 12 '18 at 0:06
  • $\begingroup$ @mroman I have used 'data augmentation' in the past such as rotation/adding noise/etc as given by DL frameworks. However, just having 3 or 11 sample images to start with is really low, isn't it? $\endgroup$ – Mona Jalal Nov 12 '18 at 0:06
  • $\begingroup$ @usεr11852 also, you say within-class accuracy is a bit irrelevant. Do you mean "per-class accuracy" such as reported in confusion matrix is irrelevant? what accuracy metric do you suggest? $\endgroup$ – Mona Jalal Nov 12 '18 at 0:08
1
$\begingroup$

There are several methods to counter imbalanced datasets. Yet, with so few samples in several of the classes I'm not sure if you will ever get a satisfactory classification. I'm not sure if you can learn class features from only 3 samples as in class 1. In any case, this might lead to extreme overfitting or at least not a good generalization.

Nevertheless, you can try the following methods:

  • Upsampling, i.e. simply doubling items from the minority classes

  • Data augmentation (as mentioned in the comments), i.e. rotating, scaling, cropping, flipping,... the images (you have to decide which modifications make sense with your application)

  • Penalizing / cost sensitive learning / class weights, i.e. penalizing the miss-classification of a minority class item more severely than that of a majority class

Disclaimer: I'm not that familiar with Pytorch as I myself work with Keras and Tensorflow most of the time. Maybe someone else can provide some Pytorch specific examples or tutorials. In any way, the keywords should at least guide you into the right direction.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.