Backpropagation vs Genetic Algorithm for Neural Network training I've read a few papers discussing pros and cons of each method, some arguing that GA doesn't give any improvement in finding the optimal solution while others show that it is more effective. It seems GA is generally preferred in literature (although mostly people modify it in some way to achieve results they need), then why majority of software solutions seem to use backpropagation only?
Is there some general rule of thumb when to use one or another? Maybe it depends on type of NN or there exists some state of the art solution which generally outperforms others?
If possible I'm looking for general answers: i.e., "if the NN is huge, GA is better", or "GA is always better but has computational performance issues" etc...
 A: One of the key problems with neural networks is over-fitting, which means that algorithms that try very hard to find a network that minimises some criterion based on a finite sample of data will end up with a network that works very well for that particular sample of data, but which will have poor generalisation.  I am rather wary of using GAs to design neural networks for this reason, especially if they do architecture optimisation at the same time as optimising the weights.  I have generally found that training networks (with regularisation) from a number (say 20) of random initial weight vectors and then forming an ensemble of all the resulting networks is generally as good an approach as any.
Essentially optimisation is the root of all evil in machine learning, the more of it you do, the more likely you are to end up over-fitting the data.
A: Whenever you deal with huge amounts of data and you want to solve a supervised learning task with a feed-forward neural network, solutions based on backpropagation are much more feasible. The reason for this is, that for a complex neural network, the number of free parameters is very high. One industry project I am currently working on involves a feed-forward neural network with about 1000 inputs, two hidden layers @ 384 neurons each and 60 outputs. This leads to 1000*384 + 384*384 + 384*60 = 554496 weight parameters which are to be optimized. Using a GA approach here would be terribly slow. 
A: Second answer is wrong. Overfitting isn't caused by optimization. Overfitting happens when your model is over-complicated and can fit all the datapoints without learning the actual rule that created them (i.e. just memorizing them, in the extreme case.) There are many ways to prevent overfitting such as choosing simpler models, dropout, dropconnect, weight decay, and just using more data. The goal should be to optimize your network and make it as accurate as possible, taking those constraints into account.
To answer the question, backprop is supposedly much faster than stochastic optimization (genetic algorithms and the like.) My guess is this is because it takes advantage of what the actual output was supposed to be, adjusts the weights in the right direction based on that, where stochastic optimization tries completely random changes and ignores that information.
However by exploring a larger area, GAs will probably do better in the long run by avoiding local optimas, it will just take longer to train.
I am curious how much slower GAs are than backprop, and if anyone knows of hybrid algorithms (scatter search seems like it would be ideal for this.)
A: If you look carefully at the scientific literature you'll find contrasting results. Obviously, in some cases GA (and more in general, Evolutionary Algorithms) may help you to find an optimal NN design but normally they have so many drawbacks (algorithm parameters' tuning, computational complexity etc) and their use is not feasible for real-world applications. Of course you can find a set of problems where GA/EAs is always better than backpropagation. Given that finding an optimal NN design is a complex multimodal optimization problem GA/EAs may help (as metaheuristics) to improve the results obtained with "traditional" algorithms, e.g. using GA/EAs to find only the initial weights configuration or helping traditional algorithms to escape from local minima (if you are interested I wrote a paper about this topic). 
I worked a lot on this field and I can tell you that there are many scientific works on GA/EAs applied to NNs because they are (or better, they used to be) an emerging research field. 
A: imho the difference between GA and backpropagation is that GA is based on random numbers and that backpropagation is based on a static algorithm such as stochastic gradient descent. GA being based on random numbers and add to that mutation means that it would likely avoid being caught in a local minima. But then GA being based on random numbers means that it is fairly likely for 2 different times you run the learning on the same network, it may reach a different conclusion i.e. a different set of weights 
A: To answer your question, I have tried to write selective algo to train mnist without tensorflow or pytorch. Only using numpy. It works but terribly slow.   So tried utilizing gpu using cupy. I optimized my code. I wrote a custom kernel to cupy etc etc ...
So I have invented binary minist :P . Very small dataset only with ones and zeros. And tried GA algorithm with this. It worked
The solution I have reached is backpropagation math and code applicaiton is far easier to understand and track to see what is going on. Selective algorithms are easier to coder harder to track (what is going on how it evolves)
Main part of code:

My results:

