In calculus 101 we learned about how to optimize a function using "analytical method": we just need to get the derivative of cost function and set the derivative to 0 then solve the equation. This is really a toy problem and will almost never happen in real world.
In real world, many cost functions are not have derivative everywhere (Further, the cost function may be discrete and do not have any derivative at all). In addition, even you can calculate the derivative, you cannot just solve the equation analytically (for example, think about how to solve $x^7+x^3-5^2+e^x+log(x+x^2)+1/x=0$ analytically? I can tell you the numerical answer is $x=1.4786$, but do not know analytical solution). We must use some numerical methods (check why here on polynomial cases Abel Ruffin Theorem).
Iterative methods are great to use, and very intuitive to understand. Suppose you want to optimize one function, instead of solving an equation and get the answer, you try to improve your answer by number of iterations /steps after enough iteration, you will get the answer close to "true answer". Say if you use calculus to minimize $f(x)=x^2$, you directly get $x=0$, but using numerical methods, you may get $x=1.1234\times10^{-20}$.
Now, it is important to understand how these iterative methods work. The key concept is knowing how to update your input parameters to get a better solution. Suppose you want to minimize $f(x_1,x_2)=x_1^2+x_2^2+|x_1+x_2|$ (note this cost function is not differentiable everywhere, but differentiable have at "most places", this is good enough for us, since we know how to update at "most places".), currently you are at $(1,1)$, and the cost is $4.0$, now you want to update $(x_1,x_2)$ to make objective function smaller. How would you do that? You may say I want to decrease both $x_1$ $x_2$, but why? In fact you are implicit using the concept of gradient "changing small amount of $x$, what will happen on $y$".. In $(1,1)$, the derivative is $(3,3)$, so negative gradient times a learning rate say $\alpha=0.001$, is $(-0.003,-0.003)$, so we updated our solution from $1, 1$ to $(0.997, 0.997)$ which have better cost.