Why do we always illustrate and depict the Loss Functions of Neural Networks as Non-Convex?
By doing a quick Google Images Search of "Loss Functions for Neural Networks" - we are generally shown the same types of pictures. The Loss Function is generally illustrated as a colorful and wavy/bumpy surface :
My Question: Does anyone know why we often use these kinds of illustrations? My impression is that we want to point out that the Loss Functions for Neural Networks are typically Non-Convex and characterized by such features as "Saddle Points" and "(Several) Local Minimums" - reinforcing the idea that Loss Functions in Neural Networks generally have complicated structures and behaviors, making them difficult for optimization algorithms to minimize and ultimately difficult to train.
Is this assertion correct - Are the Loss Functions of Neural Networks generally illustrated as "Non Convex Functions", because they actually tend to be "Non Convex" in real life? Can we ever mathematically show that the Loss Functions of Neural Networks tend to be Non Convex in real life? (E.g. Sigmoidal Activation Functions can be shown to be Non-Convex, thus Loss Functions that are comprised of Sigmoidal Activation Functions are also Non-Convex?)
Can someone please comment on this?
Thanks!
