Gradient descent - why the partial derivative?

Question

I'm quite new to AI/ML, and I was learning about gradient descent. I saw this equation that explained the gradient descent algorithm:

I quite understood everything except the reason this equation uses the partial derivative of the cost function with respect to θ_j. The instructor I was following said that the derivative is used to find the lowest value for the cost function J(θ₀θ₁).

Why is this used? How does it minimize θ_j?

Answer 1

You use a vector of partial derivatives

also known as the gradient.

In vector form the equation is

$$\begin{bmatrix}\theta_0 \\ \theta_1 \end{bmatrix} := \begin{bmatrix}\theta_0 \\ \theta_1 \end{bmatrix} - \alpha\begin{bmatrix}\frac{\partial}{\partial \theta_0} \\ \frac{\partial}{\partial \theta_1} \end{bmatrix} J(\theta_0,\theta_1) $$

---

Path along the slope of a surface

The gradient is the direction along which the function has the largest increase (and you take a step $-\alpha$ in opposite direction).

With the descent algorithm, you take steps down the slope,

• each coordinate is updated according to it's derivative
• effectively that is like following the direction of the gradient.

Below is an example image from this question. The image shows how the gradient descent follows a path along the slope of the function, moving down to the minimum value.

I have placed on top of it some extra arrows near the first step in the top. These arrows show the first step can be decomposed into two components, one for each coordinate. These steps are the single derivatives that you have in your equation.

Answer 2

Try to imagine that our cost function J(θ0,θ1) having this shape

As you can see the function has a bunch of local minima and an absolute minima (BLUE) as well ass the local and the absolute maximum (RED) .

The goal of the algorithm of gradient descent is to change the θi so that the cost is minimized as far as we can go but remember there is the over fitting phenomena that you need to avoid that happening , for more information about over-fitting visit this Source .

Now if we compute the derivative of that function with respect to θi the result will be a vector that contains the slope of the function

Now if you subtract that gradient multiplied by alpha witch is the learning rate ( the length of our J'(θ1,θ2) vector or we can say the step that the you take downhill ) from the θi the next time we compute the cost it will and should be less than the previous one .

We repeat the gradient descent algorithm until we get approximately a null partial derivative that means that we are in the minima of the cost function .

You can find much more information Here