Skip to main content
added 5 characters in body
Source Link

It should not.

[1] has shown that gradient descent with random initialization and appropriate constant step size does not converge to a saddle point. It is a long discussion but to give you an idea of why see the following example:

$$f(x,y)=\frac12 x^2+ \frac14y^4 - \frac12y^2$$

enter image description here

The critical points are $$z_1=\begin{bmatrix}0\\0\end{bmatrix}, z_2=\begin{bmatrix}0\\1\end{bmatrix}, z_3=\begin{bmatrix}0\\-1\end{bmatrix}$$.

The points $z_2$ and $z_3$ are isolated local minima, and $z_1$ is a saddle point.

Gradient descent initialized from any point of the form $z_0=\begin{bmatrix}x\\0\end{bmatrix}$ converges to the saddle point $z_1$. Any other initial point either diverges or converges to a local minimum, so the stable set of $z_1$ is the $x$-axis, which is a zero measure set in $\mathbb{R}^2$. By computing the Hessian, $$\nabla^2f(x,y)=\begin{bmatrix}1&&0\\0&&3y^2-1\end{bmatrix}$$

we find that $\nabla^2f(z_1)$ has one positive eigenvalue with eigenvector that spans the $x$-axis, thus agreeing with our above characterization of the stable set. If the initial point is chosen randomly, there is zero probability of initializing on the $x$-axis and thus zero probability of converging to the saddle point $z_1$.

It should not.

[1] has shown that gradient descent with random initialization and appropriate constant step size does not converge to a saddle point. It is long discussion but to give you an idea why see the following example:

$$f(x,y)=\frac12 x^2+ \frac14y^4 - \frac12y^2$$

enter image description here

The critical points are $$z_1=\begin{bmatrix}0\\0\end{bmatrix}, z_2=\begin{bmatrix}0\\1\end{bmatrix}, z_3=\begin{bmatrix}0\\-1\end{bmatrix}$$.

The points $z_2$ and $z_3$ are isolated local minima, and $z_1$ is a saddle point.

Gradient descent initialized from any point of the form $z_0=\begin{bmatrix}x\\0\end{bmatrix}$ converges to the saddle point $z_1$. Any other initial point either diverges or converges to a local minimum, so the stable set of $z_1$ is the $x$-axis, which is a zero measure set in $\mathbb{R}^2$. By computing the Hessian, $$\nabla^2f(x,y)=\begin{bmatrix}1&&0\\0&&3y^2-1\end{bmatrix}$$

we find that $\nabla^2f(z_1)$ has one positive eigenvalue with eigenvector that spans the $x$-axis, thus agreeing with our above characterization of the stable set. If the initial point is chosen randomly, there is zero probability of initializing on the $x$-axis and thus zero probability of converging to the saddle point $z_1$.

It should not.

[1] has shown that gradient descent with random initialization and appropriate constant step size does not converge to a saddle point. It is a long discussion but to give you an idea of why see the following example:

$$f(x,y)=\frac12 x^2+ \frac14y^4 - \frac12y^2$$

enter image description here

The critical points are $$z_1=\begin{bmatrix}0\\0\end{bmatrix}, z_2=\begin{bmatrix}0\\1\end{bmatrix}, z_3=\begin{bmatrix}0\\-1\end{bmatrix}$$.

The points $z_2$ and $z_3$ are isolated local minima, and $z_1$ is a saddle point.

Gradient descent initialized from any point of the form $z_0=\begin{bmatrix}x\\0\end{bmatrix}$ converges to the saddle point $z_1$. Any other initial point either diverges or converges to a local minimum, so the stable set of $z_1$ is the $x$-axis, which is a zero measure set in $\mathbb{R}^2$. By computing the Hessian, $$\nabla^2f(x,y)=\begin{bmatrix}1&&0\\0&&3y^2-1\end{bmatrix}$$

we find that $\nabla^2f(z_1)$ has one positive eigenvalue with eigenvector that spans the $x$-axis, thus agreeing with our above characterization of the stable set. If the initial point is chosen randomly, there is zero probability of initializing on the $x$-axis and thus zero probability of converging to the saddle point $z_1$.

Source Link

It should not.

[1] has shown that gradient descent with random initialization and appropriate constant step size does not converge to a saddle point. It is long discussion but to give you an idea why see the following example:

$$f(x,y)=\frac12 x^2+ \frac14y^4 - \frac12y^2$$

enter image description here

The critical points are $$z_1=\begin{bmatrix}0\\0\end{bmatrix}, z_2=\begin{bmatrix}0\\1\end{bmatrix}, z_3=\begin{bmatrix}0\\-1\end{bmatrix}$$.

The points $z_2$ and $z_3$ are isolated local minima, and $z_1$ is a saddle point.

Gradient descent initialized from any point of the form $z_0=\begin{bmatrix}x\\0\end{bmatrix}$ converges to the saddle point $z_1$. Any other initial point either diverges or converges to a local minimum, so the stable set of $z_1$ is the $x$-axis, which is a zero measure set in $\mathbb{R}^2$. By computing the Hessian, $$\nabla^2f(x,y)=\begin{bmatrix}1&&0\\0&&3y^2-1\end{bmatrix}$$

we find that $\nabla^2f(z_1)$ has one positive eigenvalue with eigenvector that spans the $x$-axis, thus agreeing with our above characterization of the stable set. If the initial point is chosen randomly, there is zero probability of initializing on the $x$-axis and thus zero probability of converging to the saddle point $z_1$.