Epoch vs Incremental Update over Entire Dataset In the context of neural networks, what is the difference between training like this:
foreach observation in dataset:
    compute_error(observation)
    update_weights()

and
foreach observation in dataset:
    compute_error(observation)
update_weights()

Both train over the entire dataset.
 A: If I understand the difference correctly, if you're stuck in a local minimum the first method may not be able to get you out. The weight change at each increment would be too small.
A: The difference is huge. The first method is called "stochastic", "stochastic gradient descent" (SGD) or "online" training, while the other one is typically referred to as "batch". The most commonly used one is actually a hybrid, "mini batch training", which calculates the updates on a small set of samples--depending on your application, it ranges from 5 to 10 to 100 or 1000.
Let's take a step back. In neural networks, you are performing an optimization of a loss
$$\mathcal{L}(\theta, \{x_i\}_{i=1}^N)$$
over a set of $N$ training samples $\{x_i\}_{i=1}^N$ with respect to your parameters $\theta$. To do so, you require the gradient 
$${\partial \mathcal{L(\theta, \{x_i\}_{i=1}^N)} \over \partial \theta}.$$
This is the correct way to do it, but if your data set is large, this quantity is costly to compute: it requires a complete pass over the data. Instead, one typically resorts to estimating the gradient, e.g.
$$g_i = {\partial \mathcal{L(\theta, \{x_i\})} \over \partial \theta}).$$
Note that we fixed $i$ here to a single instance.
It turns out, that following $g_i$ to lead us to the minimum, we only need $g_i$ to be correct in the expectation, i.e.
$$\mathbb{E}[g_i] = {\partial \mathcal{L(\theta, \{x_i\}_{i=1}^N)} \over \partial \theta}.$$
This is the case for iid samples.
There are many more intuitive but less formal arguments for SGD. For example, it is believed to be less prone to local minima. On the other hand, it also has regularizatory effects, e.g. overfits less than for example using LBFGS.
