Actually I am struggling for a long time with this problem and had tried a lot of experiments.
I am working on an object detection model using the SSD architecture with various backbones. I started and experimented with a lot of hyperparameters with a resnet34 backbone. Training loss curve seems to be okay in almost all the case but validation loss starts to increase even with
weight_decay = 0.1, which is really high value.
voc_map on train_set comes out to be more than 90% but on val_set the maximum which I have achieved using resnet34 is 35%. I see that model is heavily overfitting. I have used different augmentations as well.
Another weird thing about the loss curve which I observed is that, generally,
validation classification loss starts to increase early and after training on more epochs
validation localization loss also starts to increase.
Some other details:
Optimizers tried: adam, amsgrad, sgd, adamw scheduler tried: cyclic lr scheduler with triangular2 mode architectures tried: resnet101, resnet50, resnet34, resnet18, vgg16 Classification loss: Cross-Entropy with Hard Negative Mining Localization Loss: Smooth L1 loss weight decay tried: 0.001, 0.01, 0.05, 0.1, 0.2 lr tried: various values from 1e-5 to 3e-3 total_loss: 2*localization_loss + classification_loss (various ratios like 1:2, 2:1, 1:3 is tried here as well)
I have attached various losses for one of the experiment and it has the almost same pattern in all the experiments.