Neural nets: How to get the gradient of the cost function from the gradient evaluated for each observation? The gradient of the cost function, $E$, changes for each input observation. I have taken $E$ to be the sum of least squares error, for example. To see this, note that the partial derivative with respect to the weight connecting neuron $j$ in layer $l-1$ and neuron $i$ in layer $l$ is given by 
$$
\frac{\partial E}{ \partial w_{ij}^{(l)} } = \frac{\partial E}{\partial z_i^{(l)}} x_j^{(l)},
$$
where $x_j^{(l)}$ is the input to neuron $j$ in layer $l$. Therefore I will get a weight estimate for each input to the layer if I were to use gradient decent with this gradient.
How do extend what I've found to get the best weights for the whole training set, and not only for each individual observation? Is it correct to use gradient descent for each training example separately, then take the average of each weight at the end? Or should I take the average of each gradient at each iteration in gradient decent? Or something else? 
EDIT: I am NOT making the statement that $E$ changes with each training observation. But it seems that the gradient should.
 A: The statement that $E$ changes for each input observation is correct only for Stochastic Gradient Descent, where you are making updates for each sample individually (i.e. with a batch size of $1$). 
If you are performing SGD with any batch size, then you could theoretically average the gradients of the batches. If you want to be $100\%$ correct and the batch size doesn't divide the training set perfectly, then you could perform a weighted average and give the last batch's cost a contribution relative to its size.
If you are performing Gradient Descent (i.e. you can fit the whole training set in memory), then $E$ is the cost for the whole dataset.
A: If you are training deep (>3 hidden layers) neural nets, I would advise you to use stochastic gradient descent, even if you can fit your whole dataset into a single batch.
Rough SGD loop:


*

*Calculate gradient for each example in a batch (I recommend 32 examples).

*Average (mean) the gradient across examples in the batch.

*Change the weights in the direction of the gradient a little (learning rate), usually between 1e-3 and 1e-2.

*Pick a new batch and go to step 1.


After you repeat this enough times (at least 1 epoch, probably more than 10) your loss will stabilise.
There are some tricks to make it work better:


*

*You should also randomise the order in which you pick examples for good results.

*Add exponential averaging of gradient across batches (momentum).


The reason for using SGD as opposed to plain Gradient Descent, is that if you gradient descent on the whole dataset you are likely to end up in a minimum which is specific to your dataset, but is not true in population (over-fitting).
EDIT: Another important thing to remember: you need to initialize your weights randomly or your network might train poorly. In case of fully connected layers, if they have the same weights, then the gradients will be equal and all nodes be the same.
