I'd hesitate to say that it's "a well established fact" that there are lots of saddle points in neural network (and other high-dimensional) modeling problems, but the arguments from the paper you cited are compelling to me.
There have also been a number of other, related papers that have come out in the past few years; here are a few, newest first:
"Equilibrated adaptive learning rates for non-convex optimization."
Yann N. Dauphin, Harm de Vries, Yoshua Bengio
This paper (also by Bengio's group) discusses learning algorithms that explicitly attempt to regularize the diagonal of the inverse Hessian, and presents equilibrated SGD, which is another algorithm in this vein.
"Qualitatively characterizing neural network optimization problems."
Ian J. Goodfellow, Oriol Vinyals, Andrew M. Saxe
This paper shows that in several different types of common neural network problems, a line search between initial starting conditions and final, optimized parameter values often does not reveal any local minima at all. There appear to be many, small, local minima near the "bottom" of the loss that are all more or less equivalent in performance.
"ADADELTA: An Adaptive Learning Rate Method."
Matthew D. Zeiler
ADADELTA, like equilibrated SGD and RMSprop, is another learning algorithm that attempts to regularize the diagonal of the Hessian, although it does so in a very different way from ESGD. There isn't much discussion of local minima in this paper, but it's an interesting piece of work related to the development of learning algorithms that scale the learning rate based on the inverse Hessian.