How to speed up training of a Neural Network? I'm writing a thesis where I developed a script that generates NN and precalculates weights and biases to reduce a required number of epochs when I train a network. In my work, using examples I managed to prove the efficiency of precalculated weights and biases, but wondering, is there any other ways to reduce a required number of epochs to train a network? I just want to enrich my thesis, if you can share a reference I would really appreciate it
I am using feedforward and recurrent NN, applying backpropagation and stochastic gradient descent optimization
 A: This is on of the famous problems among the deep learning community. There are two solutions that I have come across so far.


*

*Deep Networks with Stochastic Depth (https://arxiv.org/abs/1603.09382)


This paper talks about a training method where you train only a set of randomly chosen layers and drop the rest with identity function. This method also works as a regularizer to avoid overfitting of the model.


*FreezeOut: Accelerate Training by Progressively Freezing Layers (https://arxiv.org/abs/1706.04983)


This paper proposes to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass.
A: If you are using SGD, there's certainly easy ways to speed up training.  You can see that SGD is the slowest of all of momentum, NAG, Adagrad, Adadelta, and RMSprop.  Here is the result of an animation by Alec Radford. See this page for more.

Also you can increase the learning rate in SGD.
A: Try the Adam optimizer. Adaptive optimizers are in fashion these days.
They converge faster as they "adapt" to the gradient and loss updates. 

Also, try other weight initialization schemes. They are also important for learning. The latest ones I know are Xavier initialization and He initialization.
Finally, also test various activation functions. I haven't had a change to look at it in detail, but SELU (scaled exponential linear units) is SOTA at the moment. There are numerous choices in the activation function as well, e.g., Relu, Elu, leaky Relu, maxout etc. 
There are a lot of choices. Experiment with many of them! 
A: in addition to the other answers:
Batchnorm by Szegedy speeds up accuracy. Plus usually a slight improvement in accuracy.
You can check also a very similar concept called Layernorm. And check out Dropout. Another technique to actually prevent overfitting, but it should also speed up your training.
