2
$\begingroup$

I'm quite new to AI/ML, and I was learning about gradient descent. I saw this equation that explained the gradient descent algorithm: equation

I quite understood everything except the reason this equation uses the partial derivative of the cost function with respect to θj. The instructor I was following said that the derivative is used to find the lowest value for the cost function J(θ0θ1).

Why is this used? How does it minimize θj?

$\endgroup$
3
  • 2
    $\begingroup$ Are you wondering why the gradient descent algorithm leads you to a local minimum (under certain conditions of course)? $\endgroup$
    – gunes
    Nov 6, 2021 at 19:30
  • $\begingroup$ Partial derivative instead of what? // Have you taken (multivariable) calculus? $\endgroup$
    – Dave
    Nov 6, 2021 at 22:35
  • $\begingroup$ Yep, even though it's a long while ago $\endgroup$
    – Robo
    Nov 7, 2021 at 5:03

2 Answers 2

3
$\begingroup$

You use a vector of partial derivatives

also known as the gradient.

In vector form the equation is

$$\begin{bmatrix}\theta_0 \\ \theta_1 \end{bmatrix} := \begin{bmatrix}\theta_0 \\ \theta_1 \end{bmatrix} - \alpha\begin{bmatrix}\frac{\partial}{\partial \theta_0} \\ \frac{\partial}{\partial \theta_1} \end{bmatrix} J(\theta_0,\theta_1) $$


Path along the slope of a surface

The gradient is the direction along which the function has the largest increase (and you take a step $-\alpha$ in opposite direction).

With the descent algorithm, you take steps down the slope,

  • each coordinate is updated according to it's derivative
  • effectively that is like following the direction of the gradient.

Below is an example image from this question. The image shows how the gradient descent follows a path along the slope of the function, moving down to the minimum value.

I have placed on top of it some extra arrows near the first step in the top. These arrows show the first step can be decomposed into two components, one for each coordinate. These steps are the single derivatives that you have in your equation.

example

$\endgroup$
1
  • $\begingroup$ Thanks! A really in-detail explanation of the sort I was hoping for! $\endgroup$
    – Robo
    Nov 7, 2021 at 5:06
1
$\begingroup$

Try to imagine that our cost function J(θ0,θ1) having this shape enter image description here

As you can see the function has a bunch of local minima and an absolute minima (BLUE) as well ass the local and the absolute maximum (RED) .

The goal of the algorithm of gradient descent is to change the θi so that the cost is minimized as far as we can go but remember there is the over fitting phenomena that you need to avoid that happening , for more information about over-fitting visit this Source .

Now if we compute the derivative of that function with respect to θi the result will be a vector that contains the slope of the function

Now if you subtract that gradient multiplied by alpha witch is the learning rate ( the length of our J'(θ1,θ2) vector or we can say the step that the you take downhill ) from the θi the next time we compute the cost it will and should be less than the previous one .

We repeat the gradient descent algorithm until we get approximately a null partial derivative that means that we are in the minima of the cost function .

You can find much more information Here

$\endgroup$
1
  • $\begingroup$ Thank you for explaining this concept in simple terms! $\endgroup$
    – Robo
    Nov 7, 2021 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.