The gradient of the cost function, $E$, changes for each input observation. I have taken $E$ to be the sum of least squares error, for example. To see this, note that the partial derivative with respect to the weight connecting neuron $j$ in layer $l-1$ and neuron $i$ in layer $l$ is given by $$ \frac{\partial E}{ \partial w_{ij}^{(l)} } = \frac{\partial E}{\partial z_i^{(l)}} x_j^{(l)}, $$ where $x_j^{(l)}$ is the input to neuron $j$ in layer $l$. Therefore I will get a weight estimate for each input to the layer if I were to use gradient decent with this gradient.

How do extend what I've found to get the best weights for the whole training set, and not only for each individual observation? Is it correct to use gradient descent for each training example separately, then take the average of each weight at the end? Or should I take the average of each gradient at each iteration in gradient decent? Or something else?

EDIT: I am NOT making the statement that $E$ changes with each training observation. But it seems that the gradient should.

  • $\begingroup$ In my opinion - you are describing batching. If you set the batch to the size of the input - then you are averaging the gradient of each input for a specific weight. I would not call that the best weight tho - In contrary for nns it is experimental found that small batch updates work better. $\endgroup$
    – Edv Beq
    Mar 23, 2020 at 1:48

2 Answers 2


The statement that $E$ changes for each input observation is correct only for Stochastic Gradient Descent, where you are making updates for each sample individually (i.e. with a batch size of $1$).

If you are performing SGD with any batch size, then you could theoretically average the gradients of the batches. If you want to be $100\%$ correct and the batch size doesn't divide the training set perfectly, then you could perform a weighted average and give the last batch's cost a contribution relative to its size.

If you are performing Gradient Descent (i.e. you can fit the whole training set in memory), then $E$ is the cost for the whole dataset.

  • $\begingroup$ Thank you for your answer. I am doing gradient decent, and what you say is clear from the definition of $E$. But are you saying that it is false that $\partial E / \partial w_{ij}^{(k)}$ changes with each $\vec x^{(1)}$ from the training data? $\endgroup$
    – Mikkel Rev
    Mar 23, 2020 at 0:31
  • $\begingroup$ $E$ is the average cost for all $x$ in the training set. Each sample would have its own cost, but in GD you average them to get a single number, i.e. $E$. $\endgroup$
    – Djib2011
    Mar 23, 2020 at 0:35
  • 1
    $\begingroup$ @Djib2011 Pretty sure you are wrong, in any non-trivial case the gradient would depend both on the input and on the weights. $\endgroup$
    – Rizhiy
    Mar 23, 2020 at 0:52
  • 1
    $\begingroup$ @Rizhiy of course it depends on the inputs. What I'm saying is that in GD you average the cost for all inputs to compute the gradient. $\endgroup$
    – Djib2011
    Mar 23, 2020 at 9:05
  • 1
    $\begingroup$ @MikkelRev if you're referring to $x_j^{(l)}$ then this, as you stated before is the input to layer $l$. This is meant to propagate the cost further back. You can theoretically go back to $l=1$ and compute the derivative of the cost $E$ to the input layer, but this isn't done in Gradient Descent, because you don't care about the gradient w.r.t the input, only the weights (because those are the ones you need to update). Fun fact: the gradient of the cost w.r.t the input is computed in other applications such as style transfer, neural dreaming and interpretability, but not in GD. $\endgroup$
    – Djib2011
    Mar 23, 2020 at 12:11

If you are training deep (>3 hidden layers) neural nets, I would advise you to use stochastic gradient descent, even if you can fit your whole dataset into a single batch.

Rough SGD loop:

  1. Calculate gradient for each example in a batch (I recommend 32 examples).
  2. Average (mean) the gradient across examples in the batch.
  3. Change the weights in the direction of the gradient a little (learning rate), usually between 1e-3 and 1e-2.
  4. Pick a new batch and go to step 1.

After you repeat this enough times (at least 1 epoch, probably more than 10) your loss will stabilise.

There are some tricks to make it work better:

  • You should also randomise the order in which you pick examples for good results.
  • Add exponential averaging of gradient across batches (momentum).

The reason for using SGD as opposed to plain Gradient Descent, is that if you gradient descent on the whole dataset you are likely to end up in a minimum which is specific to your dataset, but is not true in population (over-fitting).

EDIT: Another important thing to remember: you need to initialize your weights randomly or your network might train poorly. In case of fully connected layers, if they have the same weights, then the gradients will be equal and all nodes be the same.

  • $\begingroup$ Thanks, I will use SGD then. But I still need to understand GD. Are you able to add a paragraph about the statement that «the gradient changes for every input observation»? That is that the equation in question shows. Thanks! $\endgroup$
    – Mikkel Rev
    Mar 23, 2020 at 0:53
  • $\begingroup$ If the input is different, then the gradients will most likely be different as well. For plain Gradient Descent you calculate individual gradients for each sample and average them. $\endgroup$
    – Rizhiy
    Mar 23, 2020 at 0:54
  • $\begingroup$ If you would add the details of that process with a couple of equations in a paragraph in your writeup, I will accept it as the final answer. $\endgroup$
    – Mikkel Rev
    Mar 23, 2020 at 0:56
  • $\begingroup$ Not really sure, which equations you need. Here is more on gradient descent and SGD: en.wikipedia.org/wiki/Gradient_descent, en.wikipedia.org/wiki/Stochastic_gradient_descent $\endgroup$
    – Rizhiy
    Mar 23, 2020 at 1:06
  • $\begingroup$ I don’t need any, I understand what you’re saying, but that’s the point of the site right? To make an archive of explanations. Some summations showing that you average all the gradients in each iteration goes a long way. $\endgroup$
    – Mikkel Rev
    Mar 23, 2020 at 1:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.