Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective function to adapt the neural network parameters.
However, we know that Taylor expansions can be used to give a precise equation between a function and a power series of its $n$-order derivatives at a single point.
So shouldn’t it be theoretically possible to use the local information in a single batch of datapoints point to make a global estimate of the form of the objective function as a function of the parameters?
I am not saying of course that this inference somehow give us certainty about the optimal parameters, but shouldn’t we be able to at least use the information of second, third, ...., $n$’th order derivatives to more efficiently descend to a good parameter vector?