I try to implement logistic regression with auto-correcting learning rate and I am puzzled by the outcome.
At some point the cost of the function gets bigger than previously (to focus on some numbers let's say 628, when previously was 78). So I undo this step and at the same time decrease the learning rate from 0.297 to 0.148. And I compute the cost again -- this time 92. So I undo this step as well and decrease the learning rate to 0.074.
I do computation once again and the result is -- 106.
And there is where I am puzzled at. One possibility is my algorithm has somewhere a bug, the other is the learning rate has another purpose -- because I don't see how decreasing the learning rate (step) can possibly lead to increase of the function cost.
My workflow is such:
- compute derivative of cost function
- decrease $ \theta $ (I hope this is meaningful) vector by the above multiplied by learning rate factor
- compute cost
And since I am just starting, I perform 20 steps, just for testing the algorithm.