I am training a system that combines two sub-systems: one for classification and another for reconstruction. Can anyone suggestion what are the common practice for weight selection for combining two losses? The numerical values of two losses for first ten iterations are as follows:

Loss 1 (Cross entropy):

Epoch 1: 1.163

Epoch 2: 0.307

Epoch 3: 0.208

Epoch 4: 0.158

Epoch 5: 0.128

Epoch 6: 0.108

Epoch 7: 0.092

Epoch 8: 0.082

Epoch 9: 0.073

Epoch 10: 0.067

Loss 2 (MSE):

Epoch 1: 26.433

Epoch 2: 4.691

Epoch 3: 0.306

Epoch 4: 0.043

Epoch 5: 0.039

Epoch 6: 0.036

Epoch 7: 0.035

Epoch 8: 0.034

Epoch 9: 0.034

Epoch 10: 0.033


Long story short, there is no algorithm to do that for you, you have to utilize a heuristic approach. How you could do it without getting deep into the domain knowledge and desired loss dynamics:

  1. Running the evaluation on the untrained model to get the mean value of sub-losses.
  2. Normalizing the sub-losses according to the means you got from the previous step.

This has worked for me for a variety of problems when I wanted a quick solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.