Global optimization of deep learning models

Question

Apparently, deep neural networks have been making an impact recently. The layer-by-layer training of these networks has made it feasible to construct complex, deep, and well-performing neural networks. Still, I feel that some applications of deep learning models might benefit from global optimization approaches that do not easily get stuck in local optima. I haven't seen any research in this direction, though.

Couldn't evolutionary/biologically inspired algorithms (e.g., Particle Swarm Optimization, Differential Evolution) be used to make deep learning models more powerful? Or is the computing power necessary for this particular combination of techniques currently a limiting factor?

Answer 1

In general, gradient based techniques for optimizing neural networks are more specific and optimized for the task than the two generic optimization algorithms you mention, which don't require a gradient. Geoff Hinton mentioned evolution based approaches to optimizing neural networks in his slides on deep learning. He says that they don't really work, and they scale poorly to networks that have many weights. Using the gradient to optimize helps immensely to do efficient training.

Successful approaches to training deep neural networks have gone in the direction of approximating the second derivative of the objective function.

I am very skeptical that general optimization procedures that don't know about the structure of the neural nets are going to have much success.

Answer 2

Probably it's less researched subject at the moment as multipoint search algorithms require more processing power than using the gradient (usually).

Multipoint search algorithms do converge to a better optimum though. You can also use e.g. evolutionary algorithms the fllowing ways:

1. optimize number of layers, number of neurons and meta parameters of the network, which is an open question at the moment and traditionally requires much human interaction.

2. build a multipoint search algorithm where other than crossover and mutation, a new evolutionary operator is the one based on the gradient

3. evolve good starting point for the gradient search algorithm