# Neural nets: How to get the gradient of the cost function from the gradient evaluated for each observation?

The gradient of the cost function, $$E$$, changes for each input observation. I have taken $$E$$ to be the sum of least squares error, for example. To see this, note that the partial derivative with respect to the weight connecting neuron $$j$$ in layer $$l-1$$ and neuron $$i$$ in layer $$l$$ is given by $$\frac{\partial E}{ \partial w_{ij}^{(l)} } = \frac{\partial E}{\partial z_i^{(l)}} x_j^{(l)},$$ where $$x_j^{(l)}$$ is the input to neuron $$j$$ in layer $$l$$. Therefore I will get a weight estimate for each input to the layer if I were to use gradient decent with this gradient.

How do extend what I've found to get the best weights for the whole training set, and not only for each individual observation? Is it correct to use gradient descent for each training example separately, then take the average of each weight at the end? Or should I take the average of each gradient at each iteration in gradient decent? Or something else?

EDIT: I am NOT making the statement that $$E$$ changes with each training observation. But it seems that the gradient should.

• In my opinion - you are describing batching. If you set the batch to the size of the input - then you are averaging the gradient of each input for a specific weight. I would not call that the best weight tho - In contrary for nns it is experimental found that small batch updates work better. Mar 23 '20 at 1:48

The statement that $$E$$ changes for each input observation is correct only for Stochastic Gradient Descent, where you are making updates for each sample individually (i.e. with a batch size of $$1$$).

If you are performing SGD with any batch size, then you could theoretically average the gradients of the batches. If you want to be $$100\%$$ correct and the batch size doesn't divide the training set perfectly, then you could perform a weighted average and give the last batch's cost a contribution relative to its size.

If you are performing Gradient Descent (i.e. you can fit the whole training set in memory), then $$E$$ is the cost for the whole dataset.

• Thank you for your answer. I am doing gradient decent, and what you say is clear from the definition of $E$. But are you saying that it is false that $\partial E / \partial w_{ij}^{(k)}$ changes with each $\vec x^{(1)}$ from the training data? Mar 23 '20 at 0:31
• $E$ is the average cost for all $x$ in the training set. Each sample would have its own cost, but in GD you average them to get a single number, i.e. $E$. Mar 23 '20 at 0:35
• @Djib2011 Pretty sure you are wrong, in any non-trivial case the gradient would depend both on the input and on the weights. Mar 23 '20 at 0:52
• @Rizhiy of course it depends on the inputs. What I'm saying is that in GD you average the cost for all inputs to compute the gradient. Mar 23 '20 at 9:05
• @MikkelRev if you're referring to $x_j^{(l)}$ then this, as you stated before is the input to layer $l$. This is meant to propagate the cost further back. You can theoretically go back to $l=1$ and compute the derivative of the cost $E$ to the input layer, but this isn't done in Gradient Descent, because you don't care about the gradient w.r.t the input, only the weights (because those are the ones you need to update). Fun fact: the gradient of the cost w.r.t the input is computed in other applications such as style transfer, neural dreaming and interpretability, but not in GD. Mar 23 '20 at 12:11

If you are training deep (>3 hidden layers) neural nets, I would advise you to use stochastic gradient descent, even if you can fit your whole dataset into a single batch.

Rough SGD loop:

1. Calculate gradient for each example in a batch (I recommend 32 examples).
2. Average (mean) the gradient across examples in the batch.
3. Change the weights in the direction of the gradient a little (learning rate), usually between 1e-3 and 1e-2.
4. Pick a new batch and go to step 1.

After you repeat this enough times (at least 1 epoch, probably more than 10) your loss will stabilise.

There are some tricks to make it work better:

• You should also randomise the order in which you pick examples for good results.