Linked Questions

25 votes
1 answer

Sum or average of gradients in (mini) batch gradient decent? [duplicate]

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online ...
danijar's user avatar
  • 990
6 votes
1 answer

Cross entropy versus Mean of Cross Entropy [duplicate]

In many neural network applications, people are prone to define loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels,logits) [tensorflow functions] ...
Logan's user avatar
  • 252
3 votes
1 answer

Loss reduction: when to use sum and when mean? [duplicate]

In the PyTorch documentation for most losses, there is a parameter called reduction usually, and it is mean, but there is also a ...
Alex's user avatar
  • 31
5 votes
0 answers

Why divide the learning rate by the size of the mini batch? [duplicate]

In Michael Nielsen's online book Neural Networks and Deep Learning, in chapter one (and onwards) he divides the learning rate, $\eta$, by the size of the mini batch when he performs stochastic ...
Ulf Aslak's user avatar
  • 396
2 votes
1 answer

Sum versus mean of loss function in neural networks [duplicate]

For training a neural network is there any significant difference between using the Sum Squared error or the Mean squared error as the loss function?
Gabi23's user avatar
  • 143
74 votes
4 answers

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

First of all, I realized if I need to perform binary predictions, I have to create at least two classes through performing a one-hot-encoding. Is this correct? However, is binary cross-entropy only ...
infomin101's user avatar
  • 1,743
29 votes
2 answers

Is it common practice to minimize the mean loss over the batches instead of the sum?

Tensorflow has an example tutorial about classifying CIFAR-10. On the tutorial the average cross entropy loss across the batch is minimized. ...
Clash's user avatar
  • 773
6 votes
3 answers

Impose a condition on neural network

I am building a neural network model with TensorFlow and Keras in python. My model is performing well on unseen data in the way I desire and everything is fine. but the problem that I don't have any ...
Inevitable's user avatar
1 vote
1 answer

Batch gradient descent in Perceptron linear classifier

I'm learning about batch gradient descent for the Perceptron linear classifier and I'm confused about the update rule. On Wikipedia, it says that the update rule for batch gradient descent is $w := w -...
Learning's user avatar
  • 131
0 votes
1 answer

How does Tensorflow compute SparseCategoricalCrossentropy?

The expression for categorical cross-entropy loss can be obtained via the negative log likelihood. In particular, let $C$ be the number of mutually exclusive classes in a multi-class classification ...
Dmitry Zotikov's user avatar
2 votes
2 answers

Neural language model training - stochastic vs batch

Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer. Previously ...
Marek's user avatar
  • 143
2 votes
0 answers

How to perfectly overfit neural network to memorize input?

Say I have 15 data points (x values), for which I have corresponding y values (say randomly generated). For learning purposes, I am trying design a neural network which perfectly matches the input ...
NeverStopLearning's user avatar