I'm trying to understand the differences between the update rule for stochastic gradient ascent and descent. I've read some articles and still don't understand how to calculate the update rule:

Gradient Descent: $$\frac{∂NLL(θ)}{∂θ_j} = (hθ(x(i))-y(i))x(i)$$

So the update rule uses a minus sign because we want to minimize the log-likelihood: $$θ_j:= θ_j−α(\frac{∂NLL(θ)}{∂θ_j}) = θ_j−α[(hθ(x(i))-y(i))x(i)]$$

Gradient Ascent: $$\frac{∂LL(θ)}{∂θ_j} =(y(i) - hθ(x(i)))x(i)$$

So the update rule uses a minus sign because we want to maximize the log-likelihood: $$θ_j:= θ_j+α(\frac{∂LL(θ)}{∂θ_j}) = θ_j+α[(hθ(x(i))-y(i))x(i)]$$

Now, if I take the -1 from the derivative of the NLL I end up with the same equation as for the update rule for the gradient ascent:

$$θ_j:= θ_j−α(\frac{∂NLL(θ)}{∂θ_j}) = θ_j−α(-1)[(-hθ(x(i))+y(i))x(i)] = θ_j+α[(y(i)-hθ(x(i)))x(i)]$$

So, I know I'm wrong as they shouldn't be the same right? I'll appreciate your help understanding this.

You used $$- \alpha (-1)$$. You need to pick one, either you use $$- \alpha$$ or $$+ \alpha(-1)$$.
They should be the same. Maximizing function $$f$$ is the same as minimizing $$-f$$. Gradient ascent of $$f$$ is the same as gradient descent of $$-f$$.
• ok, so for the update rule in gradient descent I have to use either $−α(\frac{∂LL(θ)}{∂θ_j})$ or $+α(\frac{∂NLL(θ)}{∂θ_j})$ right? Sep 16, 2022 at 18:14
• @CrisTina maximizing a function $f$ is the same as minimizing $-f$.