How to combine two losses of different magnitudes such that they're weighted equally?

Question

I am training a neural network for multiclass classification.

I want to combine a custom loss function with another more generic loss. I've been simply adding it, and it works quite well but towards the end the though the custom loss is weighted more heavily because it has a bigger magnitude.

I've done something like loss1(loss2/thresh) where thresh is just the threshold I want loss2 to be less than. Which was worse than just adding it.

It's suboptimal and was hoping someone here might point me in a better direction.

Best Regards,

Oliver

Answer 1

The usual course of action is having a parameter $\lambda$ to balance the losses, e.g. $L=L_1+\lambda L_2$, and treat $\lambda$ as a hyper-parameter. They won't be always equally balanced, but this way, you can choose the importance you want to give to each loss.