Why is two-sided gradient checking more accurate? [closed]

Question

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:

One-sided difference:

$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta)}{\epsilon}$

Two-sided difference:

$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta - \epsilon)}{2\epsilon}$

Where $\epsilon$ is a small value $\epsilon \approx 10^{-4}$ (Numerical issues may arise if it is too small.)

Professor Ng asserts that the two-sided difference gives a better approximation of the true gradient.

Why is this the case?

Answer 1

One way to look at this is by Taylor approximation. Remember $$f(x+\Delta x)\approx f(x)+\Delta x f'(x)+\frac 1 2 \Delta x^2 f''(x)+\frac 1 6 \Delta x^3f'''(x)+\dots$$

One sided looks like this $$\frac{f(x+\Delta x)-f(x)}{\Delta x}\approx f'(x)+\frac 1 2 \Delta x f''(x)$$

Two sided looks like this $$\frac{f(x+\Delta x)-f(x-\Delta x)}{2\Delta x}\approx f'(x)+\frac 1 6 \Delta x^2 f'''(x)$$

In other words the two sided plays on the symmetry, and cancels out the contribution of the square term in Taylor expansion. So, when you squeeze $\Delta x$, the two sided's error diminishes much quicker than one sided.

Answer 2

For a theoretical analysis you need to read a book on numerical analysis. But intuitively it seems reasonable to appriximate the tangent line with a secant between two point symmetric about the point of tangency. Lets look at a simple numerical example, let $f(x)=x^2$ and we are interested in the derivative at $x_0=0$, so we know the true value is zero. Then the symmetrical two-sided difference gives $$ \frac{f(\epsilon)-f(-\epsilon)}{\epsilon}=\frac{(\epsilon)^2-(-\epsilon)^2}{\epsilon}=0 $$ for all $\epsilon \not =0$. Withe the asymmetric difference you can see for yourself that the value will be either $\epsilon$ or $-\epsilon$. Just as an illustration.