1
$\begingroup$

In gradient descent algorithm, the update rule of vector parameter is as follow:

From this formula, i think that the update rule only depends on the sign of the gradient. So why don't we just use arbitrary vector instead of gradient vector in this formula.

$\endgroup$
2
  • 1
    $\begingroup$ So you know how far to step in each direction. Different dimensions may need less or more changes. $\endgroup$ May 19 at 7:29
  • $\begingroup$ @AryaMcCarthy Sorry can you make it rigorously ? It doesn't seem make sense to me. $\endgroup$
    – Huy Nguyen
    May 19 at 7:41
1
$\begingroup$

At a given point, gradient is the direction of steepest increase. So, the negative of it will be the steepest decrease direction and, by definition, gradient descent algorithm uses that direction to minimise the function.

If you use an arbitrary vector, you could end up choosing $-\nabla f$ as well, and increase the value of the function.

$\endgroup$
1
$\begingroup$

The goal of using a gradient descent approach is minimizing a certain function; that is, to reach the lowest possible point on the curve of that function.

So imagine you are at the top of a mountain (or at any point on the hill for that matter) and you want to reach the bottom of the mountain. At each step, you choose a direction in which to move in order to reach the bottom of the mountain. Now, the fastest (think, optimal) way of reaching the bottom, is to always follow the steepest way down. This way, you can roll fast and will (in general) always be sure of reaching the bottom. This is the idea behind following the opposite direction of the gradient at each point, since this will be the direction of the steepest descent.

Now, of course, if you are hiking in the mountain, you can choose instead to walk in an arbitrary direction at each step, and hope that someday you will reach the bottom of the mountain. It may or may not work, but it definitely is not the optimal way of reaching the bottom. That's why, we do not use an arbitrary vector to update our parameters.

$\endgroup$
2
  • 2
    $\begingroup$ "the fastest (think, optimal) way of reaching the bottom, is to always follow the steepest way down." This is not true. The fastest way would be to step in the direction of the minimum, which is generally different than the steepest direction. The direction of the minimum is of course unknown ahead of time. But, better choices than the steepest direction can often be made, so gradient descent can often be beaten by other optimization algorithms. $\endgroup$
    – user20160
    May 19 at 11:01
  • 1
    $\begingroup$ @user20160 You are right, actually. I think I have simplified the problem for the sake of explanation, but my statement is too strong indeed. I will correct it. Thank you :) $\endgroup$
    – Jakinduria
    May 20 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.