In deep learning book (8th chapter, section 8.2.7), while explaining the challenges in neural nets optimization, the authors argued that currently researchers are focusing on finding the good initial points of parameters. They wrote that:

Many existing research directions are aimed at finding good initial points for problems that have difficult global structure, rather than at developing algorithms that use nonlocal moves.

I do not understand what does it mean by difficult global structure? How can we evaluate if the problem that neural nets are going to solve have difficult global structure? Moreover, I do not understand what do they mean by 'developing algorithms that use nonlocal moves'?


Convexity is an example of "global structure" because it depends on the value of a function everywhere, whereas (sub)differentiability is a aspect of local structure because a function is often differentiable in a limited domain.

A nonlocal move is in contrast to the commonly used SGD algorithm which operates by iteratively adding some small delta to a point in parameter space -- each update is "local" in the sense that there are no big jumps. A genetic algorithm which combines and mutates weights from a population would consist of nonlocal moves, since parameters can jump arbitrarily in the space.

There has only been very limited success with neural networks with non-SGD based, "nonlocal" algorithms. On the other hand, just by using proper initialization, we have been able to train 10000 layer neural networks.

  • $\begingroup$ By the way, do all evolutionary algorithms work in same way? Like they can throw at any point in parameter space? $\endgroup$ – samra irshad Sep 7 '18 at 23:40
  • 1
    $\begingroup$ @samrairshad No, it depends on the specifics of the algorithm. The recent "evolutionary strategies" algorithm performs only local moves. $\endgroup$ – shimao Sep 7 '18 at 23:49
  • $\begingroup$ I see. What is the reason for genetic algorithms or other evolutionary strategies being not successful in training deep neural nets? Computational intensiveness? $\endgroup$ – samra irshad Sep 7 '18 at 23:55
  • 1
    $\begingroup$ @samrairshad Possibly because genetic algorithms tend to not make use of the valuable gradient information, therefore they have to work much harder to compete with algorithms which do, such as SGD. But I'm not an expert in optimization. $\endgroup$ – shimao Sep 8 '18 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.