4
$\begingroup$

I see some implementation minimize, let's say, $\sum_{i=1}^n ||\hat{x}_i - x_i|| +\frac{1}{2}\lambda||w||^2$, I had also saw those that minimize $\frac{1}{n}\sum_{i=1}^n ||\hat{x}_i - x_i|| +\frac{1}{2}\lambda||w||^2$.

Which way is preferable? I can see that they are mathematically equivalent if tuning $\lambda$ and assume we always use all the example in training. But I am not sure about batch training.

A tensorflow version of this question would be, should we apply tf.reduce_mean or tf.reduce_sum when calculating the loss?

$\endgroup$
2
$\begingroup$

You are right that if you always use the same number examples per minibatch then it doesn't matter if the loss is a sum or a mean. However, as you observed, when the loss is a sum, if you change the batch size you have to make a corresponding change to the learning rate. It's easier to get the hyperparameter tuning right if you use the mean for the loss because you can tune batch size and learning rate independently.

$\endgroup$
1
$\begingroup$

Another benefit I find is that the loss is more easy to inteperate. The loss you'll see per iteration, excluding the regularization part, is essentially the "average" loss of each data point. This might be useful as an easy way to find out how well your model might actually work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.