0
$\begingroup$

I try to implement logistic regression with auto-correcting learning rate and I am puzzled by the outcome.

At some point the cost of the function gets bigger than previously (to focus on some numbers let's say 628, when previously was 78). So I undo this step and at the same time decrease the learning rate from 0.297 to 0.148. And I compute the cost again -- this time 92. So I undo this step as well and decrease the learning rate to 0.074.

I do computation once again and the result is -- 106.

And there is where I am puzzled at. One possibility is my algorithm has somewhere a bug, the other is the learning rate has another purpose -- because I don't see how decreasing the learning rate (step) can possibly lead to increase of the function cost.

Update

My workflow is such:

  • compute derivative of cost function
  • decrease $ \theta $ (I hope this is meaningful) vector by the above multiplied by learning rate factor
  • compute cost

And since I am just starting, I perform 20 steps, just for testing the algorithm.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You should say something about your learning procedure between the changes in the learning rate. If you are running a fixed number of steps, then a smaller learning rate can mean you travel less distance toward the optimum. It is different if you run until the cost plateaus. $\endgroup$ – Douglas Zare Sep 23 '12 at 18:43
  • $\begingroup$ @DouglasZare, updated, however your answer indicates that learning rate is just that -- learning rate, and if observe that with decreasing learning rate (from the same reference point of $\theta$) cost increases, there is a bug in my algorithm. $\endgroup$ – greenoldman Sep 23 '12 at 21:14
  • 1
    $\begingroup$ If your cost function increases as you train, your learning rate may be too high. If your cost function increases when you train with a low learning rate, and you aren't doing something like stochastic gradient descent, then you probably have a bug. $\endgroup$ – Douglas Zare Sep 23 '12 at 21:37
1
$\begingroup$

As Douglas Zare wrote, indeed it was a bug -- in my case the undo of $\theta$ didn't work correctly. So all in all learning rate vs. cost is good indicator if something goes wrong.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.