Why is two-sided gradient checking more accurate? In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:
One-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta)}{\epsilon}$
Two-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta - \epsilon)}{2\epsilon}$
Where $\epsilon$ is a small value $\epsilon \approx 10^{-4}$ (Numerical issues may arise if it is too small.)
Professor Ng asserts that the two-sided difference gives a better approximation of the true gradient.
Why is this the case?
 A: One way to look at this is by Taylor approximation. Remember 
$$f(x+\Delta x)\approx f(x)+\Delta x f'(x)+\frac 1 2 \Delta x^2 f''(x)+\frac 1 6 \Delta x^3f'''(x)+\dots$$
One sided looks like this 
$$\frac{f(x+\Delta x)-f(x)}{\Delta x}\approx f'(x)+\frac 1 2 \Delta x f''(x)$$
Two sided looks like this 
$$\frac{f(x+\Delta x)-f(x-\Delta x)}{2\Delta x}\approx f'(x)+\frac 1 6 \Delta x^2 f'''(x)$$
In other words the two sided plays on the symmetry, and cancels out the contribution of the square term in Taylor expansion. So, when you squeeze $\Delta x$, the two sided's error diminishes much quicker than one sided.
A: For a theoretical analysis you need to read a book on numerical analysis. But intuitively it seems reasonable to appriximate the tangent line with a secant between two point symmetric about the point of tangency. Lets look at a simple numerical example, let $f(x)=x^2$ and we are interested in the derivative at $x_0=0$, so we know the true value is zero. Then the symmetrical two-sided difference gives 
$$
\frac{f(\epsilon)-f(-\epsilon)}{\epsilon}=\frac{(\epsilon)^2-(-\epsilon)^2}{\epsilon}=0
$$
for all $\epsilon \not =0$.  Withe the asymmetric difference you can see for yourself that the value will be either $\epsilon$ or $-\epsilon$. Just as an illustration.
