23k views

### Sum or average of gradients in (mini) batch gradient decent? [duplicate]

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
• 990
5k views

### Cross entropy versus Mean of Cross Entropy [duplicate]

In many neural network applications, people are prone to define loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels,logits) [tensorflow functions] ...
• 252
9k views

### Loss reduction: when to use sum and when mean? [duplicate]

In the PyTorch documentation for most losses, there is a parameter called reduction usually, and it is mean, but there is also a ...
• 31
248 views

### Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...
• 396
245 views

### Sum versus mean of loss function in neural networks [duplicate]

For training a neural network is there any significant difference between using the Sum Squared error or the Mean squared error as the loss function?
• 143
145k views

### Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only ...
• 1,743
29k views

### Is it common practice to minimize the mean loss over the batches instead of the sum?

Tensorflow has an example tutorial about classifying CIFAR-10. On the tutorial the average cross entropy loss across the batch is minimized. ...
• 773
3k views

### Impose a condition on neural network

I am building a neural network model with TensorFlow and Keras in python. My model is performing well on unseen data in the way I desire and everything is fine. but the problem that I don't have any ...
• 113
1 vote
918 views