As is known to all, stochastic gradient descent is a popular optimizer in machine learning. There have been many variants of SGD. However, it has come to my attention that no one talks about the conjugate version of SGD.(i.e. incorporate 'conjugate' ideas with stochastic settings)
In general, it's impossible to maintain conjugacy of search directions in stochastic settings. But I think just keeping some of the conjugacy does not sound like a bad idea.
I googled it and found only a few related papers with low citations. I'm wondering if there is something wrong with stochastic conjugate gradient descent. Anybody any ideas?

  • 1
    $\begingroup$ It is talked about in one of my books on neural networks from the 90's (either bishop or haykin). I think it is complex, and SGD with momentum has shown to be sufficient in many papers, even if not theoretically optimal. $\endgroup$ – Stuart Holliday Sep 10 '18 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.