# Some confusion of Gradient Descent?

In logistics regression we use Gradient Descent to minimize the error function. For example:

We have error function

e(x) = hθ(x) - y


To get an appropriate θ to minimize error(x), we use Gradient Descent.

To simplify my question lets assume θ, x, y are all normal numbers, not vectors.

So in Gradient Descent. First we get derivative of e(x)

∂e(x)/∂θ


Then we run below code in many loops until we get an appropriate θ. The step is a very small distance or shift on θ.

θ := θi - (∂e(x)/∂θ)* step
`

Here is what confused me.

(∂e(x)/∂θ)* step is a small shift on e(x) not θ. So why θ minus the shift of e(x) make sense? If θ need to minus something, I think the step is more appropriate since step is a shift of θ

Run θ := θi - step in a loop until that the (∂e(x)/∂θ) is zero can also get the right θ and it seems more reasonable

• This doesn't change your question or the answer, but should be mentioned: logistic regression minimizes the log loss, not the 'error function' $e(x)$ that you wrote. Sep 29, 2017 at 5:07

The Gradient Descent method is trying to optimize the parameter $\theta$ by minimization of loss function $e(x)$.

Its typical to initialize at some point $x_{0}$ and there is a corresponding error $e(x_{0})$. The objective of gradient descent is to update the: (i) parameter $\theta$ ; and the (ii) step-size, based on the change in error.

The core intuition (key clarification regarding your doubt) is that the change in error will gradually decrease as the algorithm approaches the desired minimum point. Consequently, the step-size is 'modulated' by the 'gradient on the error-surface' or slope on the error-surface. This also lends the method its name.

Let's however explore the point a little further. What happens if we don't bother with the gradient of the error surface and simply update the parameter by a fixed value, $\delta(e)/\delta(\theta) = \lambda$, which is a constant. What you get is something like the graph on the left, instead of the graph on the right which is what the use of gradient provides. Without gradient, the parameter update will 'overshoot' the minimum. (The algorithm's termination condition will have to be updated to larger final error tolerance or a max number of iterations). 