Actually, I am trying to learn a model using stochastic gradient descent. I am using just a small subset of my entire data at a time to make each iteration. As far as I know, stochastic gradient descent can fluctuate but still moves towards the maximum point.
I am outputting the value of the objective function and I could see that it's fluctuating between 10 and 43 and not getting above it. My assumption is even though it fluctuates it slowly keeps on approaching towards the maximum value. It's not guaranteed that it will reach the global optimum and then terminate. It can overshoot and then undershoot and then keep on fluctuating there.
But in my case it is not showing an increasing trend. At one time the value of objective function is 43. The very next time it is 11 and then 35,27. It hasn't gone above 40.
Any suggestions? What could have gone wrong?