0
$\begingroup$

I am having some issues with stochastic gradient descent. Using batch gradient descent where I consider all the training sets I have certain parameter values which I know are correct.

My function is convex,globally.

Now when I use stochastic gradient descent considering ten samples at a time, the algorithm converges nicely, but I get different parameter values. The parameter values that I get using stochastic gradient descent is like 1/2 of the one that I get using batch gradient descent.

Any insights what could go wrong?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It's probably related to the learning rate, specifically the schedule that you are using to decrease it.

Try different learning rates (common choices are 1/t, 1/sqrt(t)) and smaller convergence thresholds.

In theory, if your cost function is convex, stochastic gradient descent should take you to the global minima. But this might take several passes over your whole dataset. I would not expect it to converge to global minima with just one pass.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I have tried using different learning rates and very small convergence threshold 10^-6. But still, it converges quickly giving me parameter values that are different from the actual ones $\endgroup$
    – user31820
    Commented Aug 7, 2012 at 20:58
  • $\begingroup$ I know what the parameter values should be. Since I have got it from the batch gradient descent. I initially used very small learning rate 0.000001 as well but I got the parameter values half of the actual one. $\endgroup$
    – user31820
    Commented Aug 7, 2012 at 20:59
  • $\begingroup$ In this case, I would inspect the 'convergent' situation carefully. After SGD stops, look at the value of the batch gradient. If it is all zeros than it must be a singular point, suggesting that your cost function is not convex. Otherwise, (if the gradient is non-zero) then there has to be some data points that should take you in the right direction. $\endgroup$
    – emrea
    Commented Aug 7, 2012 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.