Try to imagine that our cost function J(θ0,θ1) having this shape
As you can see the function has a bunch of local minima and an absolute minima (BLUE) as well ass the local and the absolute maximum (RED) .
The goal of the algorithm of gradient descent is to change the θi so that the cost is minimized as far as we can go but remember there is the over fitting phenomena that you need to avoid that happening , for more information about over-fitting visit this Source .
Now if we compute the derivative of that function with respect to θi
the result will be a vector that contains the slope of the function
Now if you subtract that gradient multiplied by alpha witch is the learning rate ( the length of our J'(θ1,θ2) vector or we can say the step that the you take downhill ) from the θi the next time we compute the cost it will and should be less than the previous one .
We repeat the gradient descent algorithm until we get approximately a null partial derivative that means that we are in the minima of the cost function .
You can find much more information Here