Is it possible to train a Neural Network such that each of the last layer's nodes train with a different Loss or Objective function?

This seems possible to me. You would simply update each node's weights with gradients calculated on using different loss function. Is this intuition correct?

To simplify the problem, let us assume this is a regression problem where the loss functions are similar.

Finally, has there been an research into this? I have looked but cannot find anything on the subject. Perhaps I am not searching for the right things.

NB: The reason I ask is it would be interesting to train a model with MAE on all but the final output layer and then have 3 output nodes where the loss function as MAE and then the Pin Ball Losses for 0.9 and 0.1 quantiles. MAE is just the 0.5 quantile loss so I assume the 3 output nodes to be similar and benefit from the body of the network.


1 Answer 1


Sure, you can do that. If you sum up the gradient, you will minimize

$$\mathcal{L} = \sum_i \ell_i$$ where the $l_i$'s are your different loss functions. The more important question is–does this make sense? In some cases it does, for example if you have a probabilistic model of two factors. Then the joint log-likelihood will be a sum: $$\log p(y_1, y_2|x) = \log \left( ~p(y_1|x)p(y_2|x)~ \right) \\ = \log p(y_1|x) + \log p(y_2|x).$$

Now let $\ell_i = -\log p(y_i|x)$ et voila!

Many loss functions for neural networks, especially sum of squares and the binary cross-entropy loss are actually log likelihoods of a probabilistic model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.