1
$\begingroup$

In logistics regression we use Gradient Descent to minimize the error function. For example:

We have error function

e(x) = hθ(x) - y

To get an appropriate θ to minimize error(x), we use Gradient Descent.

To simplify my question let`s assume θ, x, y are all normal numbers, not vectors.

So in Gradient Descent. First we get derivative of e(x)

∂e(x)/∂θ

Then we run below code in many loops until we get an appropriate θ. The step is a very small distance or shift on θ.

θ := θi - (∂e(x)/∂θ)* step

Here is what confused me.

(∂e(x)/∂θ)* step is a small shift on e(x) not θ. So why θ minus the shift of e(x) make sense? If θ need to minus something, I think the step is more appropriate since step is a shift of θ

Run θ := θi - step in a loop until that the (∂e(x)/∂θ) is zero can also get the right θ and it seems more reasonable

$\endgroup$
  • 1
    $\begingroup$ This doesn't change your question or the answer, but should be mentioned: logistic regression minimizes the log loss, not the 'error function' $e(x)$ that you wrote. $\endgroup$ – user20160 Sep 29 '17 at 5:07
3
$\begingroup$

The Gradient Descent method is trying to optimize the parameter $\theta$ by minimization of loss function $e(x)$.

Its typical to initialize at some point $x_{0}$ and there is a corresponding error $e(x_{0})$. enter image description here

The objective of gradient descent is to update the: (i) parameter $\theta$ ; and the (ii) step-size, based on the change in error.

The core intuition (key clarification regarding your doubt) is that the change in error will gradually decrease as the algorithm approaches the desired minimum point. Consequently, the step-size is 'modulated' by the 'gradient on the error-surface' or slope on the error-surface. This also lends the method its name.

That answers the question.

Let's however explore the point a little further. What happens if we don't bother with the gradient of the error surface and simply update the parameter by a fixed value, $\delta(e)/\delta(\theta) = \lambda$, which is a constant. What you get is something like the graph on the left, instead of the graph on the right which is what the use of gradient provides. Without gradient, the parameter update will 'overshoot' the minimum. (The algorithm's termination condition will have to be updated to larger final error tolerance or a max number of iterations).

Of course, for complex error surfaces the gradient descent method also needs help to find the local minimum, but that is beyond the scope of this question.

enter image description here

$\endgroup$
  • $\begingroup$ Is there a proof why the difference between the two theta should be equal to the gradient times learning rate? Or is it purely based on intuition? $\endgroup$ – souparno majumder Jan 20 '18 at 18:28
0
$\begingroup$

Here's an equation that I derived to prove the relation between the change in the x-coordinate, the learning rate, and the gradient of the curve.

(due to the unavailability of the latex i have to post the picture)

enter image description here enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.