Is it common practice to minimize the mean loss over the batches instead of the sum?

Question

Tensorflow has an example tutorial about classifying CIFAR-10. On the tutorial the average cross entropy loss across the batch is minimized.

def loss(logits, labels):
  """Add L2Loss to all the trainable variables.
  Add summary for for "Loss" and "Loss/avg".
  Args:
    logits: Logits from inference().
    labels: Labels from distorted_inputs or inputs(). 1-D tensor
            of shape [batch_size]
  Returns:
    Loss tensor of type float.
  """
  # Calculate the average cross entropy loss across the batch.
  labels = tf.cast(labels, tf.int64)
  cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
      logits, labels, name='cross_entropy_per_example')
  cross_entropy_mean = tf.reduce_mean(cross_entropy, name='cross_entropy')
  tf.add_to_collection('losses', cross_entropy_mean)

  # The total loss is defined as the cross entropy loss plus all of the weight
  # decay terms (L2 loss).
  return tf.add_n(tf.get_collection('losses'), name='total_loss')

See cifar10.py, line 267.

Why doesn't it minimize the sum across the batch instead? Does it make a difference? I don't understand how this would affect the backprop calculation.

Answer 1

As mentioned by pkubik, usually there's a regularization term for the parameters that doesn't depend on the input, for instance in tensorflow it's like

# Loss function using L2 Regularization
regularizer = tf.nn.l2_loss(weights)
loss = tf.reduce_mean(loss + beta * regularizer)

In this case averaging over the mini-batch helps keeping a fixed ratio between the cross_entropy loss and the regularizer loss while the batch size gets changed.

Moreover the learning rate is also sensitive to the magnitude of the loss (gradient), so in order to normalize the result of different batch sizes, taking the average seems a better option.

---

Update

This paper by Facebook (Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour) shows that, actually scaling the learning rate according to the batch size works quite well:

Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.

which is essentially the same as to multiply the gradient by k and keep the learning rate unchanged, so I guess taking the average is not necessary.

Answer 2

I will focus on the part:

I don't understand how this would affect the backprop calculation.

First of all you've probably already noticed that the only difference between the resulting loss values is that the average loss is scaled down with respect to the sum by the factor of $\frac{1}{B}$, i.e. that $L_{SUM} = B \cdot L_{AVG}$, where $B$ is the batch size. We can easily prove that the same relation is true for a derivative of any variable wrt. the loss functions ($\frac{d L_{SUM}}{{dx}} = B \frac{d L_{AVG}}{{dx}}$) by looking at the definition of derivative: $$ \frac{dL}{{dx}} = \mathop {\lim }\limits_{\Delta \to 0} \frac{{L\left( {x + \Delta } \right) - L\left( x \right)}}{\Delta } $$ Now, we would like to multiply the value of the function and see how it affects the derivative: $$ \frac{d (c \cdot L)}{{dx}} = \mathop {\lim }\limits_{\Delta \to 0} \frac{{c \cdot L\left( {x + \Delta } \right) - c \cdot L\left( x \right)}}{\Delta } $$ When we factor out the constant and move it before the limit we should see that we come up with the original derivative definition multiplied by a constant, which is exactly what we wanted to prove: $$ \frac{d (c \cdot L)}{{dx}} = c \cdot \mathop {\lim }\limits_{\Delta \to 0} \frac{{L\left( {x + \Delta } \right) - L\left( x \right)}}{\Delta } = c \cdot \frac{d L}{{dx}} $$

In SGD we would update the weights using their gradient multiplied by the learning rate $\lambda$ and we can clearly see that we can choose this parameter in such way that the final weights updates would equal. The first update rule: $$ W := W + \lambda_1 \frac{dL_{SUM}}{dW} $$ and the second update rule (imagine that $\lambda_1 = \frac{\lambda_2}{B}$): $$ W := W + \lambda_1 \frac{dL_{AVG}}{dW} = W + \frac{\lambda_2}{B} \frac{dL_{SUM}}{dW} $$

---

The excellent finding by dontloo may suggest that using the sum might be a little bit more appropriate approach. To justify the average which seems to be more popular I'd add that using the sum might probably cause some problems with weight regularization. Tuning the scaling factor for the regularizers for different batch sizes may be just as annoying as tuning the learning rate.