I am currently a bit puzzled by how mini-batch gradient descent can be trapped in a saddle point.
The solution might be too trivial that I don't get it.
You get an new sample every epoch, and it computes a new error based on a new batch, so the cost function is only static for each batch, which means that the gradient also should change for each mini batch.. but according to this should a vanilla implementation have issues with saddle points?
Another key challenge of minimizing highly non-convex error functions common for neural networks is avoiding getting trapped in their numerous suboptimal local minima. Dauphin et al.  argue that the difficulty arises in fact not from local minima but from saddle points, i.e. points where one dimension slopes up and another slopes down. These saddle points are usually surrounded by a plateau of the same error, which makes it notoriously hard for SGD to escape, as the gradient is close to zero in all dimensions.
I would mean that especially SGD would have clear advantage against saddle points, as it fluctuates towards its convergence... The fluctuations and the random sampling , and cost function being different for each epoch should be enough reasons for not becoming trapped in one.
For full batch gradient decent does it make sense that it can be trapped in saddle point, as the error function is constant.
I am a bit confused on the two other parts.