Actually, I am trying to learn a model using stochastic gradient descent. I am using just a small subset of my entire data at a time to make each iteration. As far as I know, stochastic gradient descent can fluctuate but still moves towards the maximum point.

I am outputting the value of the objective function and I could see that it's fluctuating between 10 and 43 and not getting above it. My assumption is even though it fluctuates it slowly keeps on approaching towards the maximum value. It's not guaranteed that it will reach the global optimum and then terminate. It can overshoot and then undershoot and then keep on fluctuating there.

But in my case it is not showing an increasing trend. At one time the value of objective function is 43. The very next time it is 11 and then 35,27. It hasn't gone above 40.

Any suggestions? What could have gone wrong?

  • $\begingroup$ Could it be that you were in a region with a local maximum and happened to move into a different region with a different local maximum? It seems like that could explain the behavior. $\endgroup$ Jul 26 '12 at 18:25
  • $\begingroup$ My objective function is concave, so it should have been increasing isn't it? $\endgroup$
    – user31820
    Jul 26 '12 at 18:27

First of all, make sure you are calculating the gradient correctly. Although this sounds stupid, actually it happens to a lot of people very frequently. (It is a word from Leon Bottou's lecture!)

Secondly, in stochastic gradient descent the schedule for step size is also very important. I suspect you are using too large step sizes. Take a look at Chapter 3.4.1 (136p) of the linked book. If your step size is too large, then it may happen to not converge well. For sanity check, try very small step sizes to make sure it works OK in that case. Then you can try to tune parameters to make it faster.

  • $\begingroup$ I am calculating the gradient correctly. Also my step size is very small 0.00001. $\endgroup$
    – user31820
    Jul 26 '12 at 18:55
  • $\begingroup$ Could it be because of the data itself? $\endgroup$
    – user31820
    Jul 26 '12 at 18:56
  • $\begingroup$ Your step size being that small and your objective function fluctuating from 11 to 35 means your function is very sensitive, and maybe it is not properly scaled. Then, step size being 0.00001 still tells us nothing; it can be still too large. If you are implementing something like logistic regression then make sure your variables are normalized. $\endgroup$
    – d_ijk_stra
    Jul 26 '12 at 19:34
  • $\begingroup$ It sounds stupid, but actually I also calculated the gradient wrongly and spend hours debugging it. +1 for you 1st sentence. $\endgroup$
    – BachT
    Aug 5 '16 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.