Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex?

By doing a quick Google Images Search of "Loss Functions for Neural Networks" - we are generally shown the same types of pictures. The Loss Function is generally illustrated as a colorful and wavy/bumpy surface :

enter image description here

My Question: Does anyone know why we often use these kinds of illustrations? My impression is that we want to point out that the Loss Functions for Neural Networks are typically Non-Convex and characterized by such features as "Saddle Points" and "(Several) Local Minimums" - reinforcing the idea that Loss Functions in Neural Networks generally have complicated structures and behaviors, making them difficult for optimization algorithms to minimize and ultimately difficult to train.

Is this assertion correct - Are the Loss Functions of Neural Networks generally illustrated as "Non Convex Functions", because they actually tend to be "Non Convex" in real life? Can we ever mathematically show that the Loss Functions of Neural Networks tend to be Non Convex in real life? (E.g. Sigmoidal Activation Functions can be shown to be Non-Convex, thus Loss Functions that are comprised of Sigmoidal Activation Functions are also Non-Convex?)

Can someone please comment on this?



1 Answer 1


Because they aren’t convex. This is an example from actual empirical research I used in my other answer that tries to visualize loss landscape of an actual neural network:

Example of real-life, non-convex loss landscape. It looks like a very irregular valley in the mountains, with a lot of ups and downs, many smaller valleys and peaks. Clearly non-convex.

(source: https://www.cs.umd.edu/~tomg/projects/landscapes/ and arXiv:1712.09913)

  • $\begingroup$ @ Tim: Thank you so much for your answer! Even though this is probably a pretty obvious , I am trying to understand the math behind this : why are we so quick to say that the loss functions of neural networks are usually non-convex? out of curiosity, could we take the formal mathematical definition of convexity (e.g. provided here: en.wikipedia.org/wiki/Convex_function) and use it to establish that the loss functions of neural networks are usually non-convex? E.g. if the functional form of the loss function for a nn does not obey the convex definition - it must be non convex? thanks! $\endgroup$
    – stats_noob
    Commented Jan 23, 2022 at 6:54
  • $\begingroup$ @stats555 check the first link I mentioned. $\endgroup$
    – Tim
    Commented Jan 23, 2022 at 9:19
  • $\begingroup$ A beautiful one! $\endgroup$ Commented Jan 23, 2022 at 16:23
  • $\begingroup$ thanks everyone for your comments and answers! i have a similar question over here - can you please take a look at it if you have time? stats.stackexchange.com/questions/561522/… thank you so much! $\endgroup$
    – stats_noob
    Commented Jan 24, 2022 at 3:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.