5
$\begingroup$

New to ML and Deep Learning. I am trying my hands on some image classification using resnet50.

Here are some of the graphs I see on tensorboard:

Losses

As you can see, my total loss is mostly being influenced by the regularization loss. Is this normal? How big should the regularization loss be compared to the cross entropy loss? I know that the cross-entropy loss should ideally come down eventually during training. What about the regularization loss? It is okay for it to shoot up? What would an ideal regularization loss graph look like?

I tried the training with various parameters, and for a lot more steps as well. I can see the cross entropy loss coming down slowly, but the regularization loss is not, and hence my total loss is mostly going up.

$\endgroup$

1 Answer 1

2
$\begingroup$

Although there is no one right answer, and in the end you'll need to use hyper-parameter search to find the right amount of regularization, in general, I find it's useful to keep the regularization loss between 0.1% and 20% of the objective loss. I've never gotten good results when the regularization loss is higher than the objective loss.

Also, it doesn't look right that the regularization loss is growing so rapidly. It may be a programming bug. Are you sure that you are optimizing the objective loss + regularization loss, or are you just optimizing the objective loss? Perhaps lowering the learning rate will help, if there is no bug.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.