Linked Questions

25 votes
1 answer
23k views

Sum or average of gradients in (mini) batch gradient decent? [duplicate]

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
danijar's user avatar
  • 990
6 votes
1 answer
5k views

Cross entropy versus Mean of Cross Entropy [duplicate]

In many neural network applications, people are prone to define loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels,logits) [tensorflow functions] ...
Logan's user avatar
  • 252
3 votes
1 answer
9k views

Loss reduction: when to use sum and when mean? [duplicate]

In the PyTorch documentation for most losses, there is a parameter called reduction usually, and it is mean, but there is also a ...
Alex's user avatar
  • 31
5 votes
0 answers
248 views

Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...
Ulf Aslak's user avatar
  • 396
2 votes
1 answer
245 views

Sum versus mean of loss function in neural networks [duplicate]

For training a neural network is there any significant difference between using the Sum Squared error or the Mean squared error as the loss function?
Gabi23's user avatar
  • 143
74 votes
4 answers
145k views

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only ...
infomin101's user avatar
  • 1,743
29 votes
2 answers
29k views

Is it common practice to minimize the mean loss over the batches instead of the sum?

Tensorflow has an example tutorial about classifying CIFAR-10. On the tutorial the average cross entropy loss across the batch is minimized. ...
Clash's user avatar
  • 773
6 votes
3 answers
3k views

Impose a condition on neural network

I am building a neural network model with TensorFlow and Keras in python. My model is performing well on unseen data in the way I desire and everything is fine. but the problem that I don't have any ...
Inevitable's user avatar
1 vote
1 answer
918 views

Batch gradient descent in Perceptron linear classifier

I'm learning about batch gradient descent for the Perceptron linear classifier and I'm confused about the update rule. On Wikipedia, it says that the update rule for batch gradient descent is $w := w -...
Learning's user avatar
  • 131
0 votes
1 answer
645 views

How does Tensorflow compute SparseCategoricalCrossentropy?

The expression for categorical cross-entropy loss can be obtained via the negative log likelihood. In particular, let $C$ be the number of mutually exclusive classes in a multi-class classification ...
Dmitry Zotikov's user avatar
2 votes
2 answers
349 views

Neural language model training - stochastic vs batch

Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer. Previously ...
Marek's user avatar
  • 143
2 votes
0 answers
242 views

How to perfectly overfit neural network to memorize input?

Say I have 15 data points (x values), for which I have corresponding y values (say randomly generated). For learning purposes, I am trying design a neural network which perfectly matches the input ...
NeverStopLearning's user avatar