Transition from "old-school" neural network methods to deep learning?

Question

As far I know the current state of deep learning favours a rather simplistic setup -- in short: many layers to allow for representational learning, maxout or a similarly suited activation function to avoid the explaining-away phenomenon, and dropout as regularization. Correct me please if I'm wrong or outdated on this.

To me, having missed the early deep learning developement (RBMs, autoencoders, etc.) and still being stuck in the 90's neural network methods, this seems like a promising chance to step into the field of deep learning. It suggests I simply take my old neural network code, add maxout and dropout, use stochastic gradient descent aka backpropagation and start over (--maybe by disregarding the huge-data stuff requiring GPUs and specialized libraries). But, is it really that simple?

Questions:

• Given I want to turn a running "old-school" neural network program into a basic -- but useful -- deep-learning implementation, what do I have to do? Is it really just dropout + maxout?

• How to perform the learning? Just by stochastic backpropagation? Are there other methods used from the large set of training methods and tweaks (rprop, Levenberg-Marquart, momentum, simulated annealing, etc.)? Any preferences on the learning parameters?

• What are good heuristics for the network architecture (in terms of number of layers, number of neurons, direct-links and/or not full connections, etc.)? Yes, I know, it always depends :-) ... still, are there any reasonable approaches?

Ok, I realize this became a very broad question, parts of which are most likely already covered here on Cross Validated -- so I'm thankful for links as well.

Answer 1

The nature publication described everything precisely about what deep learning really is, which I don't have to repeat here.

And to my observation, "old school" neural network don't go "deep" for many reasons. And nowadays, people found new ways to overcome headaches brought by "old school" neural network frameworks, e.g. using GPU to improve the computational efficiency, using dropout to solve over-fitting problem etc.

• I think the answer is yes. It is those important heuristic tricks, along with the undisregardable GPU implementations that made deep learning really "deep".
• Generally, the lecture slides here answered this question clearly enough, see the last slide for a summary of learning methods for different problems.
• Recent research on deep recurrent attention models (DRAM) proved that using proper designed network models, which act more like a human brain, can improve computational efficiency (by decreasing number of parameters required to train with the help of recurrent networks), while increase recognition accuracy. This made the application of dropout less necessary for an implementation.

EDIT: The above answer may contain much opinion of my personal perspective, but truth is when we look back a few year later that how deep learning grow popular among machine learning algorithms, we would see that it is the inspiration of human brain that push the development of deep learning forward. That is to say, we are just trying to mimic how brain solve the complicated problems, while "old school" neural networks are brains like a rat, and "deep" networks are brains of a monkey.