5
$\begingroup$

In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:

One-sided difference:

$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta)}{\epsilon}$

Two-sided difference:

$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta - \epsilon)}{2\epsilon}$

Where $\epsilon$ is a small value $\epsilon \approx 10^{-4}$ (Numerical issues may arise if it is too small.)

Professor Ng asserts that the two-sided difference gives a better approximation of the true gradient.

Why is this the case?

$\endgroup$

closed as unclear what you're asking by Aksakal, S. Kolassa - Reinstate Monica, mdewey, Michael Chernick, jbowman Feb 15 '18 at 13:39

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ In what sense is "accuracy" implied here? $\endgroup$ – AdamO Feb 14 '18 at 19:17
4
$\begingroup$

One way to look at this is by Taylor approximation. Remember $$f(x+\Delta x)\approx f(x)+\Delta x f'(x)+\frac 1 2 \Delta x^2 f''(x)+\frac 1 6 \Delta x^3f'''(x)+\dots$$

One sided looks like this $$\frac{f(x+\Delta x)-f(x)}{\Delta x}\approx f'(x)+\frac 1 2 \Delta x f''(x)$$

Two sided looks like this $$\frac{f(x+\Delta x)-f(x-\Delta x)}{2\Delta x}\approx f'(x)+\frac 1 6 \Delta x^2 f'''(x)$$

In other words the two sided plays on the symmetry, and cancels out the contribution of the square term in Taylor expansion. So, when you squeeze $\Delta x$, the two sided's error diminishes much quicker than one sided.

$\endgroup$
2
$\begingroup$

For a theoretical analysis you need to read a book on numerical analysis. But intuitively it seems reasonable to appriximate the tangent line with a secant between two point symmetric about the point of tangency. Lets look at a simple numerical example, let $f(x)=x^2$ and we are interested in the derivative at $x_0=0$, so we know the true value is zero. Then the symmetrical two-sided difference gives $$ \frac{f(\epsilon)-f(-\epsilon)}{\epsilon}=\frac{(\epsilon)^2-(-\epsilon)^2}{\epsilon}=0 $$ for all $\epsilon \not =0$. Withe the asymmetric difference you can see for yourself that the value will be either $\epsilon$ or $-\epsilon$. Just as an illustration.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.