11
$\begingroup$

If we process say 10 examples in a batch, I understand we can sum the loss for each example, but how does backpropagation work in regard to updating the weights for each example?

For example:

  • Example 1 --> loss = 2
  • Example 2 --> loss = -2

This results in an average loss of 0 (E = 0), so how would this update each weight and converge? Is it simply by the randomization of the batches that we "hopefully" converge sooner or later? Also doesn't this only compute the gradient for the first set of weights for the last example processed?

$\endgroup$
14
$\begingroup$

Gradient descent doesn't quite work the way you suggested but a similar problem can occur.

We don't calculate the average loss from the batch, we calculate the average gradients of the loss function. The gradients are the derivative of the loss with respect to the weight and in a neural network the gradient for one weight depends on the inputs of that specific example and it also depends on many other weights in the model.

If your model has 5 weights and you have a mini-batch size of 2 then you might get this:

Example 1. Loss=2, $\text{gradients}=(1.5,-2.0,1.1,0.4,-0.9)$

Example 2. Loss=3, $\text{gradients}=(1.2,2.3,-1.1,-0.8,-0.7)$

The average of the gradients in this mini-batch are calculated, they are $(1.35,0.15,0,-0.2,-0.8)$

The benefit of averaging over several examples is that the variation in the gradient is lower so the learning is more consistent and less dependent on the specifics of one example. Notice how the average gradient for the third weight is $0$, this weight won't change this weight update but it will likely be non-zero for the next examples chosen which get computed with different weights.

edit in response to comments:

In my example above the average of the gradients is computed. For a mini-batch size of $k$ where we calculate the loss $L_i$ for each example we and aim to get the average gradient of the loss with respect to a weight $w_j$.

The way I wrote it in my example I averaged each gradient like: $\frac{\partial L}{\partial w_j} = \frac{1}{k} \sum_{i=1}^{k} \frac{\partial L_i}{\partial w_j}$

The tutorial code you linked to in the comments uses Tensorflow to minimize the average loss.

Tensorflow aims to minimize $\frac{1}{k} \sum_{i=1}^{k} L_i$

To minimize this it computes the gradients of the average loss with respect to each weight and uses gradient-descent to update the weights:

$\frac{\partial L}{\partial w_j} = \frac{\partial }{\partial w_j} \frac{1}{k} \sum_{i=1}^{k} L_i$

The differentiation can be brought inside the sum so it's the same as the expression from the approach in my example.

$\frac{\partial }{\partial w_j} \frac{1}{k} \sum_{i=1}^{k} L_i = \frac{1}{k} \sum_{i=1}^{k} \frac{\partial L_i}{\partial w_j}$

$\endgroup$
  • $\begingroup$ Gotcha. You would still want to average the loss over the batch_size correct? I'm not sure if you're familiar with tensorflow but I've been trying to reconcile my understanding with this tutorial: tensorflow.org/get_started/mnist/beginners You can see the loss is averaged over the batch (reduce_mean code). I suppose tensorflow keeps an internal count/averages of the weights? $\endgroup$ – carboncomputed Mar 12 '17 at 21:33
  • 1
    $\begingroup$ @carboncomputed Oh yes you're right, they average the loss so that when Tensorflow computes the gradients of the average loss it is effectively computing the average of the gradients for each loss. I'll edit my answer to show the math for this. $\endgroup$ – Hugh Mar 12 '17 at 22:11
  • $\begingroup$ Interesting. Thanks for the clarification. So just to dig a bit deeper, are the weight gradients calculated per example during the forward pass and stored or are these computed during the optimization process in tensorflow? I suppose I'm just missing "where" are these gradients in tensorflow? I see the forward pass and the loss, so tensorflow is doing these gradient calculations/averaging under the hood for me? $\endgroup$ – carboncomputed Mar 13 '17 at 3:38
  • 1
    $\begingroup$ @carboncomputed That's the appeal of Tensorflow, it uses symbolic math and can do the differentiation under the hood $\endgroup$ – Hugh Mar 13 '17 at 6:11
  • $\begingroup$ Thank you for a neat answer. However, I failed to understand how TF knows how to back propagate with an average loss as shown in this example, code line 170 ? $\endgroup$ – sinner May 8 '18 at 18:01
-1
$\begingroup$

The reason to use mini batches is to have a good amount of training example such that the possible noise of it is reduced by averaging their effects, but also it's not a full batch that for many datasets could require a huge quantity of memory. One important fact is that the error that you evaluate is always a distance between your predicted output and the real output: that means that it can't be negative, so you can't have, as you said, an error of 2 and -2 that cancel out, but it would instead become an error of 4. You then evaluate the gradient of the error with respect to all the weights, so you can compute which change in the weights would reduce it the most. Once you do so, you take a "step" in that direction, based on the magnitude of your learning rate alpha. (This is the basic concepts, I'm not going into detail about backpropagation for deep NN) After running this training on your dataset for a certain number of epochs, you can expect your network to converge if your learning step is not too big to make it diverge. You could still end up in a local minimum, this can be avoided by initializing differently your weights, using differenr optimizers, and trying to regularize.

$\endgroup$
  • $\begingroup$ Just to add: we use mini-batches mostly for computational efficiency. We have a trade-off between accuracy of descent and frequency of updating weights. Data has to be extremely large to not fit into memory. $\endgroup$ – Łukasz Grad Mar 12 '17 at 11:20
  • $\begingroup$ I understand each, but how do we update our weights for a specific batch? Are the weight gradients also summed for each example? $\endgroup$ – carboncomputed Mar 12 '17 at 11:27
  • $\begingroup$ No, there is only one gradient, which is a vector of derivatives, on the total batch error. This means that we update once our weights based on the gradient, i.e. the direction of update that makes the error on this mini batch decrease the most. The gradient is made of partial derivatives, that is the derivative pf the mini batch error with respect to each weight: this tells us if each weight should become smaller or bigger, and how much. All the weights get one update for batch, to reduce the error on that mini batch, that is independent from other mini batches. $\endgroup$ – dante Mar 12 '17 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.