6
$\begingroup$

In many neural network applications, people are prone to define loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels,logits) [tensorflow functions] as a loss function. Why add tf.reduce_mean (compute the expected value)?

| cite | improve this question | | | | |
$\endgroup$
5
$\begingroup$

For online training methods like stochastic gradient descent, the loss on each iteration reflects the contribution of a single data point. So, no summation is necessary in this case. For batch or minibatch training, it's necessary to combine the loss from each point in the batch/minibatch by taking the sum or mean.

When taking the sum, the loss depends on the number of data points (in the case of batch training) or minibatch size (in the case of minibatch training). Also note that the number of points in each minibatch may vary. Taking the mean decouples the loss from these influences.

This has a few benefits:

  1. It makes it easier to compare the loss across datasets with a different number of points, or across iterations with a different mini-batch size.

  2. It makes it possible to change the number of points or mini-batch size, without changing other parameters like step size, regularization (or penalty) strengths, etc.

  3. In the case of minibatch training, it ensures that all mini-batches contribute equally. Taking the sum would give higher weight to mini-batches containing more points.

| cite | improve this answer | | | | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.