Can we apply analyticity of a neural network to improve upon gradient descent? Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective function to adapt the neural network parameters. 
However, we know that Taylor expansions can be used to give a precise equation between a function and a power series of its $n$-order derivatives at a single point. 
So shouldn’t it be theoretically possible to use the local information in a single batch of datapoints point to make a global estimate of the form of the objective function as a function of the parameters? 
I am not saying of course that this inference somehow give us certainty about the optimal parameters, but shouldn’t we be able to at least use the information of second, third, ...., $n$’th order derivatives to more efficiently descend to a good parameter vector? 
 A: In the blog post Reflections on Kitchen Sinks, Ali Rahimi and Ben Recht conducted an experiment to assess the effectiveness of several gradient methods and Levenberg–Marquardt at solving a simple neural network: two linear fully-connected layers in a feedforward network. (The "trick" here is that the matrix they are approximating has very high condition number.)

Clearly, using LM is a huge improvement over the gradient-only methods. However, it is more expensive because it requires computing a large, dense matrix and then solving a linear system to apply a single update.
The only reason people don't use higher-order gradient information is that modern neural networks have so many parameters as to make this impractical. Re-computing the inverse Hessian (e.g. Newton's method) at each step for millions of parameters is going to be hard unless the matrix is very special (diagonal being the easiest case). But excepting the computational challenge, this can be very effective.
It would be a huge breakthrough in the field if someone could do both: (1) use higher order gradient information and (2) compute the update quickly. But this is really hard to do.
