In week 5 of Andrew Ng's Machine Learning course, he gives the formulae for gradient checking:
One-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta)}{\epsilon}$
Two-sided difference:
$\dfrac{\partial}{\partial\Theta}J(\Theta) \approx \dfrac{J(\Theta + \epsilon) - J(\Theta - \epsilon)}{2\epsilon}$
Where $\epsilon$ is a small value $\epsilon \approx 10^{-4}$ (Numerical issues may arise if it is too small.)
Professor Ng asserts that the two-sided difference gives a better approximation of the true gradient.
Why is this the case?