Multi-task learning: weight selection for combining loss functions I am training a system that combines two sub-systems: one for classification and another for reconstruction. Can anyone suggestion what are the common practice for weight selection for combining two losses? The numerical values of two losses for first ten iterations are as follows:
Loss 1 (Cross entropy): 
Epoch 1: 1.163
Epoch 2: 0.307
Epoch 3: 0.208
Epoch 4: 0.158
Epoch 5: 0.128
Epoch 6: 0.108
Epoch 7: 0.092
Epoch 8: 0.082
Epoch 9: 0.073
Epoch 10: 0.067  
Loss 2 (MSE):
Epoch 1:    26.433
Epoch 2:    4.691
Epoch 3:    0.306
Epoch 4:    0.043
Epoch 5:    0.039
Epoch 6:    0.036
Epoch 7:    0.035
Epoch 8:    0.034
Epoch 9:    0.034
Epoch 10:   0.033
 A: Long story short, there is no algorithm to do that for you, you have to utilize a heuristic approach. How you could do it without getting deep into the domain knowledge and desired loss dynamics:


*

*Running the evaluation on the untrained model to get the mean value of sub-losses.

*Normalizing the sub-losses according to the means you got from the previous step.


This has worked for me for a variety of problems when I wanted a quick solution.
A: This is a very interesting question and I had a search and found some methods:

*

*uniform combination of losses from different tasks

*dynamic weight average

*uncertainty weighting methods with various amounts of training data per-task

For the details please refer to this paper: A comparison of loss weighting strategies for multi-task learning in deepneural networks and some more up-to-date papers under the term multi-task learning loss weighting strategies or refer to this open course: CS 330: Deep Multi-Task and Meta Learning.
Hope that helps.
