5
$\begingroup$

In neural networks, I always see cost functions scaled by 1/n, where n = the total number of training examples. I also see it in things like adding a regularization term on to the cost. Why is this done?

For example, here is the cross entropy loss with a regularization term at the end enter image description here

This scaling means that the gradients for updating the weights are also scaled, so increasing your training data slows down the learning speed. If you double your training data, you will be learning at half the speed. For 2000 examples (which is nothing!) I need an insane learning rate of 1.0 just to get anywhere, which means certainly I am misunderstanding something here.

When I take out this 1/n scaling, I have no problems at all, and I don't have to adjust my learning rate when I add more training data, which makes me even more sure that I am missing something here.

Another problem is that I'm using a type of very dynamic data augmentation, where every mini batch is randomly rotated, scaled, noise added, etc. so that the network never really sees the same input twice. How would you even use this 1/n scaling in this case?

|cite|improve this question|||||
$\endgroup$
2
$\begingroup$

In terms of mini-batch learning, $n$ should be the size of the batch instead of the total amount of training data (which in your case should be infinite).

Gradients are scaled by $1/n$ because we are taking the average of the batch, so the same learning rate can be used regardless of the size of the batch.

Edit
enter image description here
I found this later in that page, which shows how to set hyper-parameters in the context of mini-batch learning. So it might seem that the formula and the paragraph you refer to are talking about full batch learning, in which case $n$ should be the total number of training data.

Edit

So now the question would become "why you use the total training data size in regularization, instead of the batch size?"

I don't know whether there's a theoretical interpretation for this, for me it is just to make the hyper-parameters stay irrelevant to the batch size, since we are not summing the regularization term across the batch.

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ This is what I thought and actually makes sense to do, but I read an article that says that n should be the total number of examples and not the batch size. I updated my question to include a link and an excerpt from it. the author seems to have a lot of great info, but do you think it's possible he was mistaken on this? $\endgroup$ – Frobot Mar 22 '16 at 19:53
  • $\begingroup$ @Frobot hi, I made some update, it seems that he was talking about full batch learning instead of mini-batch learning. $\endgroup$ – dontloo Mar 23 '16 at 2:33
  • $\begingroup$ I see where I got confused. n does change to being the value of the batch size when doing Stochastic GD, but the regularization still takes into account the number of total training examples, which made me assume the cost function did too. This is why he says you have to adjust the value of λ when you change the training data size. So now the question would become "why you use the total training data size in regularization, instead of the batch size?" $\endgroup$ – Frobot Mar 23 '16 at 5:01
  • $\begingroup$ And also I just realized that one point still remains - In non stochastic gradient descent you would still decrease how fast you learn as you increase your training examples. So this scaling still doesn't make sense to me in the context of non stochastic $\endgroup$ – Frobot Mar 23 '16 at 5:22
  • $\begingroup$ @Frobot i don't know whether there's a theoretical interpretation, for me it is just to make the hyper-parameters stay irrelevant to the batch size, and in practice $\lambda/n$ can be implemented as a single parameter. $\endgroup$ – dontloo Mar 23 '16 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.