Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Actually, I am trying to learn a model using stochastic gradient descent. I am using just a small subset of my entire data at a time to make each iteration. As far as I know, stochastic gradient descent can fluctuate but still moves towards the maximum point.

I am outputting the value of the objective function and I could see that it's fluctuating between 10 and 43 and not getting above it. My assumption is even though it fluctuates it slowly keeps on approaching towards the maximum value. It's not guaranteed that it will reach the global optimum and then terminate. It can overshoot and then undershoot and then keep on fluctuating there.

But in my case it is not showing an increasing trend. At one time the value of objective function is 43. The very next time it is 11 and then 35,27. It hasn't gone above 40.

Any suggestions? What could have gone wrong?

share|improve this question
Could it be that you were in a region with a local maximum and happened to move into a different region with a different local maximum? It seems like that could explain the behavior. –  Michael Chernick Jul 26 '12 at 18:25
My objective function is concave, so it should have been increasing isn't it? –  user31820 Jul 26 '12 at 18:27
add comment

1 Answer

First of all, make sure you are calculating the gradient correctly. Although this sounds stupid, actually it happens to a lot of people very frequently. (It is a word from Leon Bottou's lecture!)

Secondly, in stochastic gradient descent the schedule for step size is also very important. I suspect you are using too large step sizes. Take a look at Chapter 3.4.1 (136p) of the linked book. If your step size is too large, then it may happen to not converge well. For sanity check, try very small step sizes to make sure it works OK in that case. Then you can try to tune parameters to make it faster.

share|improve this answer
I am calculating the gradient correctly. Also my step size is very small 0.00001. –  user31820 Jul 26 '12 at 18:55
Could it be because of the data itself? –  user31820 Jul 26 '12 at 18:56
Your step size being that small and your objective function fluctuating from 11 to 35 means your function is very sensitive, and maybe it is not properly scaled. Then, step size being 0.00001 still tells us nothing; it can be still too large. If you are implementing something like logistic regression then make sure your variables are normalized. –  d_ijk_stra Jul 26 '12 at 19:34
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.