Some confusion of Gradient Descent?

Question

In logistics regression we use Gradient Descent to minimize the error function. For example:

We have error function

e(x) = hθ(x) - y

To get an appropriate θ to minimize error(x), we use Gradient Descent.

To simplify my question let`s assume θ, x, y are all normal numbers, not vectors.

So in Gradient Descent. First we get derivative of e(x)

∂e(x)/∂θ

Then we run below code in many loops until we get an appropriate θ. The step is a very small distance or shift on θ.

θ := θi - (∂e(x)/∂θ)* step

Here is what confused me.

(∂e(x)/∂θ)* step is a small shift on e(x) not θ. So why θ minus the shift of e(x) make sense? If θ need to minus something, I think the step is more appropriate since step is a shift of θ

Run θ := θi - step in a loop until that the (∂e(x)/∂θ) is zero can also get the right θ and it seems more reasonable

Answer 1

The Gradient Descent method is trying to optimize the parameter $\theta$ by minimization of loss function $e(x)$.

Its typical to initialize at some point $x_{0}$ and there is a corresponding error $e(x_{0})$.

The objective of gradient descent is to update the: (i) parameter $\theta$ ; and the (ii) step-size, based on the change in error.

The core intuition (key clarification regarding your doubt) is that the change in error will gradually decrease as the algorithm approaches the desired minimum point. Consequently, the step-size is 'modulated' by the 'gradient on the error-surface' or slope on the error-surface. This also lends the method its name.

That answers the question.

Let's however explore the point a little further. What happens if we don't bother with the gradient of the error surface and simply update the parameter by a fixed value, $\delta(e)/\delta(\theta) = \lambda$, which is a constant. What you get is something like the graph on the left, instead of the graph on the right which is what the use of gradient provides. Without gradient, the parameter update will 'overshoot' the minimum. (The algorithm's termination condition will have to be updated to larger final error tolerance or a max number of iterations).

Of course, for complex error surfaces the gradient descent method also needs help to find the local minimum, but that is beyond the scope of this question.

Answer 2

Here's an equation that I derived to prove the relation between the change in the x-coordinate, the learning rate, and the gradient of the curve.

(due to the unavailability of the latex i have to post the picture)