# Does learning rate have additional meaning in logistic regression?

I try to implement logistic regression with auto-correcting learning rate and I am puzzled by the outcome.

At some point the cost of the function gets bigger than previously (to focus on some numbers let's say 628, when previously was 78). So I undo this step and at the same time decrease the learning rate from 0.297 to 0.148. And I compute the cost again -- this time 92. So I undo this step as well and decrease the learning rate to 0.074.

I do computation once again and the result is -- 106.

And there is where I am puzzled at. One possibility is my algorithm has somewhere a bug, the other is the learning rate has another purpose -- because I don't see how decreasing the learning rate (step) can possibly lead to increase of the function cost.

# Update

My workflow is such:

• compute derivative of cost function
• decrease $\theta$ (I hope this is meaningful) vector by the above multiplied by learning rate factor
• compute cost

And since I am just starting, I perform 20 steps, just for testing the algorithm.

-
You should say something about your learning procedure between the changes in the learning rate. If you are running a fixed number of steps, then a smaller learning rate can mean you travel less distance toward the optimum. It is different if you run until the cost plateaus. –  Douglas Zare Sep 23 '12 at 18:43
@DouglasZare, updated, however your answer indicates that learning rate is just that -- learning rate, and if observe that with decreasing learning rate (from the same reference point of $\theta$) cost increases, there is a bug in my algorithm. –  greenoldman Sep 23 '12 at 21:14
If your cost function increases as you train, your learning rate may be too high. If your cost function increases when you train with a low learning rate, and you aren't doing something like stochastic gradient descent, then you probably have a bug. –  Douglas Zare Sep 23 '12 at 21:37
As Douglas Zare wrote, indeed it was a bug -- in my case the undo of $\theta$ didn't work correctly. So all in all learning rate vs. cost is good indicator if something goes wrong.