# Why doesn't the optimizer just look for stationary points of the loss function?

I want to have a better understanding of the weight-optimization process.

I understand the optimizer(e.g., gradient descent) looks for the direction in which to move the parameters to minimize the loss function. Than it makes a move in that direction, the size of this move is determined by the learning rate.

I have an naive question. Why don't we just look for the stationary points instead of an iterative process?

• When the dimension is even modestly large, searching for a stationary point a la grid search would be incredibly expensive computationally speaking. Oct 9, 2020 at 12:19
• How would you propose to find the stationary points of an arbitrary function without an iterative process?
– Sycorax
Oct 9, 2020 at 12:35
• Check where the derivative is 0? I am not sure that works multi-dimensional space? Oct 9, 2020 at 13:46
• Sure, functions in $\mathbb{R}^n$ have gradients. And you certainly can take the gradient and set it equal to zero. But the next step is to crank through the algebra. For some classes of functions, this is very simple. For other classes, this is harder, in the sense that the expression is grotesquely ornate, but still feasible. But we can actually show that it's impossible for still other classes of functions.
– Sycorax
Oct 9, 2020 at 14:31

Root-finding can be framed as an optimization problem because we seek to find $$x$$ such that $$f(x)=0$$; if we consider that for some polynomial function $$f$$ we are seeking a stationary point $$f^\prime(x)=0$$, then this is just root-finding for $$f^\prime$$.