Can we benefit from a convex loss function when optimizing a neural network?

Question

Many existing loss functions are convex since they are easy to optimize. However, they are only convex with respect to the output $y$, not to parameter $\theta$ of a neural network, or any other non-convex model. In this case, is it still beneficial to use a convex loss function when optimizing a neural network?

Answer 1

I don't think its affect would be noticeable if any, because the output appears only on the last layer, however the parameters goes back until the beginning and a lot of interactions take place in the intermediate layers.