Why is optimal learning rate obtained from analyzing gradient descent algorithm rarely (never) used in practice? - Cross Validated most recent 30 from stats.stackexchange.com 2019-10-21T16:04:55Z https://stats.stackexchange.com/feeds/question/191729 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/191729 3 Why is optimal learning rate obtained from analyzing gradient descent algorithm rarely (never) used in practice? user10024395 https://stats.stackexchange.com/users/79151 2016-01-21T09:25:44Z 2016-04-13T18:33:14Z <p>Why is optimal learning rate obtained from analyzing gradient descent algorithm rarely (never) used in practice? </p> <p>Gradient descent procedure is to iteratively do $a(k+1) = a(k) - \eta(k)\nabla J(a(k))$. Expanding $J(a(k+1))$ using $2^{nd}$ order Taylor expansion and taking the derivative with respect to $\eta$, one obtain the optimal learning rate of $$\eta^{opt} = \frac{||\nabla J||^2}{\nabla J^T H \nabla J}$$ where $H$ is the second order derivative of the cost function.</p> <p>However, I have not seen this being used in any learning algorithm that employs gradient descent like SVM or perceptron. Is there any reason for that? Or is it implicitly employed in a way that I am not aware of. If so, can anyone illustrate the math involved? </p> https://stats.stackexchange.com/questions/191729/-/207158#207158 7 Answer by Cliff AB for Why is optimal learning rate obtained from analyzing gradient descent algorithm rarely (never) used in practice? Cliff AB https://stats.stackexchange.com/users/76981 2016-04-13T18:02:14Z 2016-04-13T18:02:14Z <p>It's not used because it's counter productive. </p> <p>Just about the <em>only</em> justification for using gradient descent (and it's really not a good justification at all, as you will see if you read through some of the posts on the topic on this site) is that one avoids needing to calculate the Hessian, as this can be very expensive for high dimensional problems. So once you've calculated the Hessian, you've taken away gradient descent's strength: not needing to calculate the Hessian. </p> <p>If you have calculated the Hessian, you're better of using something like Newton's Method. </p> https://stats.stackexchange.com/questions/191729/-/207164#207164 0 Answer by Dikran Marsupial for Why is optimal learning rate obtained from analyzing gradient descent algorithm rarely (never) used in practice? Dikran Marsupial https://stats.stackexchange.com/users/887 2016-04-13T18:33:14Z 2016-04-13T18:33:14Z <p>There isn't much point in using it in SVM solvers or perceptrons because the cost function being minimized has properties that make other algorithms more attractive (e.g. interior points, SMO, normal equations).</p> <p>I'm not an expert on non-linear optimisation, but the equation looks rather like the Newton steps used in the line search in <a href="https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf" rel="nofollow">conjugate gradient</a> descent (but in the direction of the gradient, rather than a conjugate direction). If that is the case, it is used quite often for statistical models (such as neural networks) where the cost function does not have convenient properties that admit more efficient optimisation schemes. </p> <p>Caveat lector: My own neural network library used conjugate gradients, but I wrote it so long ago I can no longer remember the details.</p>