As far I know the current state of deep learning favours a rather simplistic setup -- in short: many layers to allow for representational learning, maxout or a similarly suited activation function to avoid the explaining-away phenomenon, and dropout as regularization. Correct me please if I'm wrong or outdated on this.
To me, having missed the early deep learning developement (RBMs, autoencoders, etc.) and still being stuck in the 90's neural network methods, this seems like a promising chance to step into the field of deep learning. It suggests I simply take my old neural network code, add maxout and dropout, use stochastic gradient descent aka backpropagation and start over (--maybe by disregarding the huge-data stuff requiring GPUs and specialized libraries). But, is it really that simple?
Given I want to turn a running "old-school" neural network program into a basic -- but useful -- deep-learning implementation, what do I have to do? Is it really just dropout + maxout?
How to perform the learning? Just by stochastic backpropagation? Are there other methods used from the large set of training methods and tweaks (rprop, Levenberg-Marquart, momentum, simulated annealing, etc.)? Any preferences on the learning parameters?
What are good heuristics for the network architecture (in terms of number of layers, number of neurons, direct-links and/or not full connections, etc.)? Yes, I know, it always depends :-) ... still, are there any reasonable approaches?
Ok, I realize this became a very broad question, parts of which are most likely already covered here on Cross Validated -- so I'm thankful for links as well.