In neural networks, I always see cost functions scaled by 1/n, where n = the total number of training examples. I also see it in things like adding a regularization term on to the cost. Why is this done?
For example, here is the cross entropy loss with a regularization term at the end
This scaling means that the gradients for updating the weights are also scaled, so increasing your training data slows down the learning speed. If you double your training data, you will be learning at half the speed. For 2000 examples (which is nothing!) I need an insane learning rate of 1.0 just to get anywhere, which means certainly I am misunderstanding something here.
When I take out this 1/n scaling, I have no problems at all, and I don't have to adjust my learning rate when I add more training data, which makes me even more sure that I am missing something here.
Another problem is that I'm using a type of very dynamic data augmentation, where every mini batch is randomly rotated, scaled, noise added, etc. so that the network never really sees the same input twice. How would you even use this 1/n scaling in this case?