13
$\begingroup$

I've been going trough Machine Learning in Action book from manning (https://www.manning.com/books/machine-learning-in-action) In the logistic regression chapter it uses gradient ascent to calculate the best weights. Why do we pick gradient ascent instead of gradient descent ?

$\endgroup$
2
  • 7
    $\begingroup$ To find a maximum rather than a minimum. $\endgroup$
    – Carl
    Jan 29, 2017 at 2:28
  • $\begingroup$ I think the different rather reflects on engineering perspective, gradient descent always uses on a minimization setting that minimum is bounded somehow (for example negative log likelihood is bounded by 0). $\endgroup$ May 21, 2020 at 1:54

4 Answers 4

22
$\begingroup$

https://en.wikipedia.org/wiki/Gradient_descent:

To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point.

If instead one takes steps proportional to the positive of the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent.

In other words:

  • gradient descent aims at minimizing some objective function: $\theta_j \leftarrow \theta_j-\alpha \frac{\partial}{\partial \theta_{j}} J(\theta)$
  • gradient ascent aims at maximizing some objective function: $\theta_j \leftarrow \theta_j+\alpha \frac{\partial}{\partial \theta_{j}} J(\theta)$
$\endgroup$
10
$\begingroup$

What is a gradient?

It is a derivative of a function at a certain point. Basically, gives the slope of the line at that point. How to calculate this slope geometrically(just consider a 2 D graph and any continuous function)? You draw a tangent at that point crossing x-axis and a perpendicular to the x-axis from that point. It will form a triangle and now calculating slope is easy. Also if this tangent is parallel to x-axis the gradient is 0 and if it is parallel to y-axis the gradient is infinity.

Why use ascent or descent?

If I have a function which is convex then at the bottom the gradient or derivative is 0. Similarly, if we have a concave function at the top gradient or derivative is 0. Why are we interested in 0? This is because it helps us find either the lowest(convex) or highest(concave) value of the function

Now our machine learning has a cost function and they can either be concave or convex. If it is convex we use Gradient Descent and if it is concave we use we use Gradient Ascent. Now there are two cost functions for logistic regression. When we use the convex one we use gradient descent and when we use the concave one we use gradient ascent. Also, note that if I add a minus before a convex function it becomes concave and vice versa.

$\endgroup$
1
$\begingroup$

@Franck is correct, they are the same thing other than the sign is positive or negative. There is good explanation in the book:

enter image description here

$\endgroup$
0
1
$\begingroup$

If you want to minimize a function, we use Gradient Descent. For eg. in Deep learning we want to minimize the loss hence we use Gradient Descent.

If you want to maximize a function, we use Gradient Ascent. For eg. in Reinforcement Learning - Policy Gradient methods our goal is to maximize the reward function hence we use Gradient Ascent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.