# Why is SSD object detection model overfitting even with very high weight decay?

Actually I am struggling for a long time with this problem and had tried a lot of experiments.

I am working on an object detection model using the SSD architecture with various backbones. I started and experimented with a lot of hyperparameters with a resnet34 backbone. Training loss curve seems to be okay in almost all the case but validation loss starts to increase even with weight_decay = 0.1, which is really high value. voc_map on train_set comes out to be more than 90% but on val_set the maximum which I have achieved using resnet34 is 35%. I see that model is heavily overfitting. I have used different augmentations as well.

Another weird thing about the loss curve which I observed is that, generally, validation classification loss starts to increase early and after training on more epochs validation localization loss also starts to increase.

Some other details:

Optimizers tried: adam, amsgrad, sgd, adamw

scheduler tried: cyclic lr scheduler with triangular2 mode

architectures tried: resnet101, resnet50, resnet34, resnet18, vgg16

Classification loss: Cross-Entropy with Hard Negative Mining

Localization Loss: Smooth L1 loss

weight decay tried: 0.001, 0.01, 0.05, 0.1, 0.2

lr tried: various values from 1e-5 to 3e-3

total_loss: 2*localization_loss + classification_loss (various ratios like 1:2, 2:1, 1:3 is tried here as well)


I have attached various losses for one of the experiment and it has the almost same pattern in all the experiments.