I am not really sure about how it behaves when using batch gradient descent in logistic regression.

As we do each iteration, $L(W)$ is getting bigger and bigger, it will jump across the largest point and $L(W)$ is going down. How do I know it without computing $L(W)$ but only knowing old $w$ vector and updated $w$ vector?

If I use regularized logistic regression, will the weights become smaller and smaller or any other patterns?


Regularization is designed to combat overfitting, but not aid in gradient descent convergence.

If you are minimizing a function $J$ parameterized by vector $\theta$ and where each element of $\theta$ is identified by $\theta_j$, (i.e. minimize $J(\theta)$).

Then the basic idea in batch gradient descent is to iterate until convergence by computing a new value of $\theta$ from the previous one in the following way. Updated each $\theta_j$ simultaneously with the formula.

$\theta_j := \theta_j - \alpha\frac{\partial }{\partial \theta_j}J(\theta)$

That $\alpha$ term is called the learning rate. It's arbitrary and if it's very small then the algorithm will converge slowly which will make the algorithm take a long time, but if it's too large, then what can happen is exactly what you are experiencing. In this case $\theta$ will be updated in the right direction, but will go too far and jump past the minimum or it can even climb out and increase.

The remedy is to simply decrease $\alpha$ until it doesn't happen. A sufficiently small learning rate guarantees that $J(\theta)$ will decrease on every iteration. The trick is to determine what value of $\alpha$ is a good one that allows fast convergence but avoids non-convergence.

A useful approach is to plot $J(\theta)$ while the algorithm is running to observe how it decreases. Start with a small value (e.g. 0.01) increase it if it appears to result in slow convergence or decrease it still further in the case of non-convergence.


I assume that $L(W)$ is the log likelihood function. The answer is convergence. When the difference between old W and updated W is small enough, you stop the iteration. Detailed implemention in R has been posted here: http://mathiology.blogspot.nl/2014/04/logistic-regression-with-gradient_7.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.