One of the most common algorithms for training Neural Networks is back propagation, which essentially does (stochastic) gradient descent on the training objective function. Gradient descent can be used to optimize any objective function, as long as we can evaluate the function and compute its (sub-)gradient at any given point in the solution space.
My question is: are there specific types of functions/layers that are believed to be hard/impossible to train in neural nets and using back propagation? Is it likely to get much better results using neural nets if we were to use more sophisticated optimization methods?
Follow up comments: I am asking this because before neural nets became epidemic the most important step in designing new models was to make sure that training the model leads to a simple optimization problem (e.g. LP, QP, convex, bi-convex, etc.); even if doing gradient descent on more complex objective functions was possible. You wouldn't just do gradient descent unless you had a customized optimization procedure (with careful initialization and what not) to do training, otherwise you would very likely get stuck in a bad local minima. But I see people becoming as creative as they can get with neural nets and throw in any function in the training objective as long as they can compute the gradient so that they can do back propagation.
Is there something special about back propagation that makes it different from gradient descent and protects it against issues like sensitivity to initialization, getting stuck in bad local minima, etc.? Or are we just excitedly enjoying the leap in the performance of neural nets, and being oblivious to those issues?