Training each Node in an NN's Output with Different Loss Function

Question

Is it possible to train a Neural Network such that each of the last layer's nodes train with a different Loss or Objective function?

This seems possible to me. You would simply update each node's weights with gradients calculated on using different loss function. Is this intuition correct?

To simplify the problem, let us assume this is a regression problem where the loss functions are similar.

Finally, has there been an research into this? I have looked but cannot find anything on the subject. Perhaps I am not searching for the right things.

NB: The reason I ask is it would be interesting to train a model with MAE on all but the final output layer and then have 3 output nodes where the loss function as MAE and then the Pin Ball Losses for 0.9 and 0.1 quantiles. MAE is just the 0.5 quantile loss so I assume the 3 output nodes to be similar and benefit from the body of the network.

Answer 1

Sure, you can do that. If you sum up the gradient, you will minimize

$$\mathcal{L} = \sum_i \ell_i$$ where the $l_i$'s are your different loss functions. The more important question is–does this make sense? In some cases it does, for example if you have a probabilistic model of two factors. Then the joint log-likelihood will be a sum: $$\log p(y_1, y_2|x) = \log \left( ~p(y_1|x)p(y_2|x)~ \right) \\ = \log p(y_1|x) + \log p(y_2|x).$$

Now let $\ell_i = -\log p(y_i|x)$ et voila!

Many loss functions for neural networks, especially sum of squares and the binary cross-entropy loss are actually log likelihoods of a probabilistic model.