Why are convex loss functions so important in Neural Networks? Because Neural networks are learnt end-to-end, with non-linear activations, causing convex loss functions to actually become non-convex problems.
So my issue is when using SGD optimizer, this requires a convex loss function, but if a Neural Network makes it non-convex, there will be multiple local optimas now, so why can we not just use non-convex functions instead to begin with? If it is not a convex problem, why does SGD work just for convex functions?
I feel that it is to do with the Hessian matrix. For example, am i right in saying that if there are more weights, the less likely it is that the Hessian is positive semidefinite, meaning most of the critical points of local optima will be saddle nodes which SGD can escape. Whereas, if it were non-convex, it is likely to be a positive semidefinite Hessian, meaning it will take longer to converge?
