1
$\begingroup$

Gradient descent uses the first order derivative information of the objective function as a function of the parameters. Gradient descent therefore uses only “local” information about the objective function to adapt the neural network parameters.

However, we know that Taylor expansions can be used to give a precise equation between a function and a power series of its $n$-order derivatives at a single point.

So shouldn’t it be theoretically possible to use the local information in a single batch of datapoints point to make a global estimate of the form of the objective function as a function of the parameters?

I am not saying of course that this inference somehow give us certainty about the optimal parameters, but shouldn’t we be able to at least use the information of second, third, ...., $n$’th order derivatives to more efficiently descend to a good parameter vector?

Improve this question
$\endgroup$
2
  • $\begingroup$ In general, iteratively forming a local, quadratic model to compute a step direction and/or size is an incredibly common technique in optimization. I'm less familiar with the particulars of training neural networks. $\endgroup$
    – Matthew Gunn
    Commented May 28, 2018 at 20:41
  • $\begingroup$ There are lots of ways to globalize a locally convergent algorithm. For example, for globalizing gradient descent. Another way is to take a global convergent algorithm, like random sampling and let it identify a surface, and then start off the gradient descent in the minimum of the surface region to speed up the otherwise painfully slow to converge random sampling method. With respect to machine learning, I suppose you could learn what the approximate surface to find the minimum for looks like, and then use gradient descent. But why do that? $\endgroup$
    – Carl
    Commented May 29, 2018 at 6:57

1 Answer 1

Reset to default
7
$\begingroup$

In the blog post Reflections on Kitchen Sinks, Ali Rahimi and Ben Recht conducted an experiment to assess the effectiveness of several gradient methods and Levenberg–Marquardt at solving a simple neural network: two linear fully-connected layers in a feedforward network. (The "trick" here is that the matrix they are approximating has very high condition number.)

enter image description here

Clearly, using LM is a huge improvement over the gradient-only methods. However, it is more expensive because it requires computing a large, dense matrix and then solving a linear system to apply a single update.

The only reason people don't use higher-order gradient information is that modern neural networks have so many parameters as to make this impractical. Re-computing the inverse Hessian (e.g. Newton's method) at each step for millions of parameters is going to be hard unless the matrix is very special (diagonal being the easiest case). But excepting the computational challenge, this can be very effective.

It would be a huge breakthrough in the field if someone could do both: (1) use higher order gradient information and (2) compute the update quickly. But this is really hard to do.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.