# Can we apply analyticity of a neural network to improve upon gradient descent? [duplicate]

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective function to adapt the neural network parameters.

However, we know that Taylor expansions can be used to give a precise equation between a function and a power series of its $n$-order derivatives at a single point.

So shouldn’t it be theoretically possible to use the local information in a single batch of datapoints point to make a global estimate of the form of the objective function as a function of the parameters?

I am not saying of course that this inference somehow give us certainty about the optimal parameters, but shouldn’t we be able to at least use the information of second, third, ...., $n$’th order derivatives to more efficiently descend to a good parameter vector?

• In general, iteratively forming a local, quadratic model to compute a step direction and/or size is an incredibly common technique in optimization. I'm less familiar with the particulars of training neural networks. May 28, 2018 at 20:41
• There are lots of ways to globalize a locally convergent algorithm. For example, for globalizing gradient descent. Another way is to take a global convergent algorithm, like random sampling and let it identify a surface, and then start off the gradient descent in the minimum of the surface region to speed up the otherwise painfully slow to converge random sampling method. With respect to machine learning, I suppose you could learn what the approximate surface to find the minimum for looks like, and then use gradient descent. But why do that?
– Carl
May 29, 2018 at 6:57