1
$\begingroup$

My convolutional network seems to work well in learning the features. However, the accuracy of the validation set is increasing very slowly with respect to the learning rate as also illustrated in the figure below:

enter image description here

The loss of both training and validation sets are shown in the figure below:

enter image description here

If I decrease the learning rate, the validation accuracy will stay around 25% and it will not increase. Is there any method to speed up the validation accuracy increment while decreasing the rate of learning?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ This looks like a case of overfitting. $\endgroup$
    – Dave
    Commented Jun 17, 2021 at 10:25
  • $\begingroup$ which framwork are you using? Keras? Pytorch? TensorFlow? somthing else? $\endgroup$
    – Jonathan
    Commented Jun 17, 2021 at 10:49
  • $\begingroup$ Pytorch @Jonathan $\endgroup$
    – gazelle
    Commented Jun 20, 2021 at 14:55

2 Answers 2

Reset to default
2
$\begingroup$

this is a classic case of overfitting - you have good results for your training set, but bad results for your validation set.

there are a few psossible things to do (the sulotion is not in the learning rate):

  1. use dropout layers, for example: conv2d->maxpool->dropout -> conv2d->maxpool->dropout
  2. use l1 regularization or l2 regularization
  3. use data augmentation / data generation: before inserting the input image to your network, apply some random transformation- rotation, strech, flip, crop, enlargement and more
  4. add more conv2d->maxpool layers

good luck!

Improve this answer
$\endgroup$
6
  • $\begingroup$ Learning rate is not totally unrelated to generalization error, a large learning rate can act as a kind of regularization, cf papers.nips.cc/paper/2019/file/… $\endgroup$
    – Jonny Lomond
    Commented Jun 17, 2021 at 11:30
  • $\begingroup$ Thank you @Jonathan. I have added all of the mentioned methods. The learning rate decreased but still, my validation accuracy is not going above 45%. After 45% accuracy, the validation loss starts to increase and its accuracy starts to decrease. Is there anything I can do about this? $\endgroup$
    – gazelle
    Commented Jun 20, 2021 at 15:01
  • $\begingroup$ Okay, lets dive into some details, the more you provide, the better we could solve it. how many images are you using in your data set? What is the percentage of images used in training/validation? What is your batch size? What is your learning rate? How many different classes do you need to classify? What is the percentage of each class from the entire dataset? What architecture /layers are you using? Which activation function are you using? How many epochs have you trained? $\endgroup$
    – Jonathan
    Commented Jun 22, 2021 at 6:28
  • $\begingroup$ @Jonathan My classifier has 4 labels. There are 1000 training images for each label and 100 validation images for each label. I have 4400 images in total. 10% validation and 90% training. The batch size is 20 and the learning rate is 0.000001. Each class has 25% of the whole dataset images. I have trained 100 epochs and the architecture is 2 layers: 1. Conv2D->ReLU->BatchNorm2D->Flattening->Dropout2D 2. Linear->ReLU->BatchNorm1D->Dropout And finally a fully connected and a softmax. $\endgroup$
    – gazelle
    Commented Jun 23, 2021 at 6:29
  • $\begingroup$ @gazelle I would suggest to change the architecture, you should have at least 3-4 conv2d layers. also Maxpool layers are usually good for classification tasks. I would suggest: [conv2d-relu-maxpool2d-dropout2d] -> [conv2d-relu-maxpool2d-dropout2d] -> [conv2d-relu-maxpool2d-dropout2d] -> [conv2d-relu-maxpool2d-dropout2d] -> flatten -> [fully connected-relu-droput1d-fully connected] -> softmaex. you can add more "blocks" of conv2d+maxpool, and see if this improves your results $\endgroup$
    – Jonathan
    Commented Jun 23, 2021 at 9:02
2
$\begingroup$

Thanks for all the comments. First, I looked at this problem as overfitting and spend so much time on methods to solve this such as regularization and augmentation. Finally, after trying different methods, I couldn't improve the validation accuracy. Thus, I went through the data. I found a bug in my data preparation which was resulting in similar tensors being generated under different labels. I generated the correct data and the problem was solved to some extent (The validation accuracy increased around 60%). Then finally I improved the validation accuracy to 90% by the technique that @Jonathan mentioned in his comment: adding more "conv2d + maxpool" layers.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.