One of the most common algorithms for training Neural Networks is back propagation, which essentially does (stochastic) gradient descent on the training objective function. Gradient descent can be used to optimize any objective function, as long as we can evaluate the function and compute its (sub-)gradient at any given point in the solution space.

My question is: are there specific types of functions/layers that are believed to be hard/impossible to train in neural nets and using back propagation? Is it likely to get much better results using neural nets if we were to use more sophisticated optimization methods?

Follow up comments: I am asking this because before neural nets became epidemic the most important step in designing new models was to make sure that training the model leads to a simple optimization problem (e.g. LP, QP, convex, bi-convex, etc.); even if doing gradient descent on more complex objective functions was possible. You wouldn't just do gradient descent unless you had a customized optimization procedure (with careful initialization and what not) to do training, otherwise you would very likely get stuck in a bad local minima. But I see people becoming as creative as they can get with neural nets and throw in any function in the training objective as long as they can compute the gradient so that they can do back propagation.

Is there something special about back propagation that makes it different from gradient descent and protects it against issues like sensitivity to initialization, getting stuck in bad local minima, etc.? Or are we just excitedly enjoying the leap in the performance of neural nets, and being oblivious to those issues?


There are plenty of functions that are hard to optimise. In fact when you look at the successful implementation, they are mainly image/signal processing using sigmoid or ReLU activation functions. (maybe you want to clarify what your target function is. I.e. do you mean the function you want to approximate or are you including the activation functions- i.e. minimising error for fixed network structure).

So a classic target function that NNs cannot approximate by gradient descent is the spiral (i.e. input is (x, y) coordinates and output is if input lies on spiral band). If you Google cascade correlation you will find pictures...

  • 1
    $\begingroup$ thank you for your answer (+1). My question/concern is more about why people not care much about training complex and non-convex neural nets using something as simple as gradient descent (back prop)? For example, in caffe.berkeleyvision.org/tutorial/layers.html you can find many different layers (there are a bunch more in their github code repository) and people have trained/used networks with much more complex layers that do, for example, geometric transformation (in the Spatial Transformers Network paper) or distance transform computation in (DPMs are CNNs paper), etc. $\endgroup$ – Sobi Dec 19 '15 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.