0
$\begingroup$
  1. I am training a model, and using the original learning rate of the author (I use their github too), I get a validation loss that keeps oscillating a lot, it will decrease but then suddenly jump to a large value and then decrease again, but never really converges as the lowest it gets is 2 (while training loss converges to 0.0 something - much below 1)

At each epoch I get the training accuracy and at the end, the validation accuracy. Validation accuracy is always greater than the training accuracy.

When I test on real test data, I get good results, but I wonder if my model is overfitting. I expect a good model's val loss to converge in a similar fashion with training loss, but this doesn't happen and the fact that the val loss oscillates to very large values at times worries me.

  1. Adjusting the learning rate and scheduler etc etc, I got the val loss and training loss to a downward fashion with less oscilliation, but this time my test accuracy remains low (as well as training and validation accuracies)

I did try a couple of optimizers (adam, sgd, adagrad) with step scheduler and also the pleateu one of pytorch, I played with step sizes etc. but it didn't really help, neither did clipping gradients.

Is my model overfitting? If so, how can I reduce the overfitting besides data augmentation? If not (I read some people on quora said it is nothing to worry about, though I would think it must be overfitting), how can I justify it? Even if I would get similar results for a k-fold experiment, would it be good enough? I don't feel it would justify the oscilliating. How should I proceed?

$\endgroup$
  • 1
    $\begingroup$ What is your definition of overfitting? Does the description you give here fit that definition? Why or why not? $\endgroup$ – Sycorax Mar 26 at 16:19
  • $\begingroup$ My definition of overfitting is having low test accuracy compared to the training accuracy. However, I would think for a model to be stable it should have a general fashion where the training and loss validation keep decreasing in a similar fashion until it converges. Some -observations- of an overfitting model would be the validation loss going up. So where does my problem sit, I am not sure, I would say it could be underfitting actually but the loss val going up and not being stable worries me, I think I confused myself by overthinking so I need a different perspective for sanity check :) $\endgroup$ – dusa Mar 27 at 8:18
1
$\begingroup$

No, it's not necessary that your model is over fitting. Check your train and validation data whether some of test data is leaked into the validation set or not. This may be one of the reasons for getting more validation accuracy than your training data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.