How to train a neural network to minimize two loss functions?

For TF/Keras (or in general), what is the best way to define a multidimensional y target? Should this even be done?

The problem:

Any sample x tries to predict several "values of interest". These values are returned as simple floats. Each value also has a corresponding category (binary) that classifies the value. Note that the binary classification corresponds to class A or B not a "yes" or "no".

y = [[1.0451 0], [1.1469, 1], [1.3571, 1], [1.0451, 0]]

Note that for my particular problem (which might be relevant to the posted problem), each y label has a different # of elements up to 5 (elements without values are zero padded [0, 0].

A possible solution:

Flatten y into a single vector. From your experience, how well do NN's establish the type of relationship between values like above?

• Do the 1.0451, 1.1469, 1.3571, and 1.0451 map to the 0, 1, 1, 0 in any way, like $>1.2\implies 1?$
– Dave
Nov 30 '21 at 18:53
• No. 1.2 can be either a 0 or 1, it really depends on the relation between 1.2 and the rest of the values in X.
– Ayma
Nov 30 '21 at 18:56
• So are you just trying to predict a continuous $y_1$ and binary $y_2$ instead of just doing the former (regression) or latter (classification)?
– Dave
Nov 30 '21 at 18:58
• Yes. I have considered splitting it into 2 separate (regression and classification) problems. However, the nature of the problem does not allow this. Hence, I need to predict both continuous y1 and binary y2 for a single x.
– Ayma
Nov 30 '21 at 19:10
• This question is on-topic here. It asks how to incorporate two targets of distinct type into a neural network. So, if we have a loss $L_1$ which is the regression portion and another loss $L_2$ which is the classification portion, how do we train a NN to minimize both? This touches on the optimization topic of multi-objective optimization, for which one approach is finding the pareto optimal solution. A simple approach would be for Ayma to choose some $0 \le \theta \le 1$ and minimize $\theta L_1 + (1 - \theta) L_2$.
– Sycorax
Nov 30 '21 at 20:59

I understand the question as asking

If we have a loss $$L_1$$ which is the regression portion and another loss $$L_2$$ which is the classification portion, how do we train a NN to minimize both?

This touches on the optimization topic of multi-objective optimization, for which one approach is finding the pareto optimal solution. A simple approach would be for Ayma to choose some $$0 \le \theta \le 1$$ and minimize $$\mathcal{L} = \theta L_1 + (1 - \theta) L_2$$.

Because $$\mathcal{L}$$ is a convex combination of $$L_1$$ and $$L_2$$, we are training a model which finds a compromise between minimizing $$L_1$$ and minimzing $$L_2$$, with the extent of the compromise chosen by $$\theta$$. Setting $$\theta = 0$$ only trains the classifier, setting $$\theta=1$$ only trains the regressor, and setting $$\theta$$ in between chooses some mix of both.

I would suggest training several models with alternative choices of $$\theta$$ and assessing which ones solve your problem best.

I can't find the reference that I'm thinking of, but sometimes having multiple targets in the same model can be helpful because the added information from the second target helps the model predict the first target better, compared to a model that has the first target as the only target. In other words, having 2 models for each of the 2 targets might be worse than using 1 model for both.

• "having 2 models ... might be worse than using 1 model for both." My thoughts exactly.
– Ayma
Nov 30 '21 at 21:34
• And it's a testable hypothesis -- you can check your intuition by training a model to predict each target singly, and compare those models to the single model with 2 targets.
– Sycorax
Nov 30 '21 at 21:36
• A problem just occurred to me. My y contains four multivariate predictions (4 pairs x 2 values = 8 total values), where every pair of values is related but the four pairs of predictions are unrelated to each other. I don't see how the pareto optimal solution would help the NN find the relation between pairs. From my understanding, it would just take all four regression values to find L1 and all four classification values to find L2.
– Ayma
Nov 30 '21 at 21:52
• Just use the sum of $\mathcal{L}$ for each of the 4 observation pairs. If this isn't what you mean, please explain what"finding the relation between the pairs" means to you.
– Sycorax
Nov 30 '21 at 21:58
• My bad, I understand perfectly. I guess I'm just searching for something that isn't really needed.
– Ayma
Nov 30 '21 at 22:09