# Different parameter values when using stochastic gradient descent

I am having some issues with stochastic gradient descent. Using batch gradient descent where I consider all the training sets I have certain parameter values which I know are correct.

My function is convex,globally.

Now when I use stochastic gradient descent considering ten samples at a time, the algorithm converges nicely, but I get different parameter values. The parameter values that I get using stochastic gradient descent is like 1/2 of the one that I get using batch gradient descent.

Any insights what could go wrong?

-
add comment

## 1 Answer

It's probably related to the learning rate, specifically the schedule that you are using to decrease it.

Try different learning rates (common choices are 1/t, 1/sqrt(t)) and smaller convergence thresholds.

In theory, if your cost function is convex, stochastic gradient descent should take you to the global minima. But this might take several passes over your whole dataset. I would not expect it to converge to global minima with just one pass.

-
I have tried using different learning rates and very small convergence threshold 10^-6. But still, it converges quickly giving me parameter values that are different from the actual ones –  user31820 Aug 7 '12 at 20:58
I know what the parameter values should be. Since I have got it from the batch gradient descent. I initially used very small learning rate 0.000001 as well but I got the parameter values half of the actual one. –  user31820 Aug 7 '12 at 20:59
In this case, I would inspect the 'convergent' situation carefully. After SGD stops, look at the value of the batch gradient. If it is all zeros than it must be a singular point, suggesting that your cost function is not convex. Otherwise, (if the gradient is non-zero) then there has to be some data points that should take you in the right direction. –  emrea Aug 7 '12 at 21:26
add comment