4
$\begingroup$

Take linear regression for instance the loss is usually define as $L = \frac{1}{2N}\sum_{i = 1}^{N}(Y_i - WX_i)^2$. Two here, as I understand, is to make the derivative look nice. But I am not sure about the role of $N$ here, I suppose it is because we are trying to look at the expected loss. However, $N$ does not really matter because a constant factor does not affect the minima.

Actually, In this notes, the loss is defined to be $\frac{1}{2}\sum_{i = 1}^{N}(Y_i - WX_i)^2$, and the motivation there is coming from maximum likelihood.

Now if we look at mini-batch gradient descent, my question is whether it is necessary to divide the mini-batch size while updating the loss function. If we are taking the perspective that $L$ is some sort of expected loss, then certainly we need to divide the mini-batch size. However, if our original loss does not divide $N$ then I do not think it even makes to even use mini-batch gradient descent because I do not really think $\sum_{i}^{M}(Y_i - WX_i)^2$ for some $M < N$ is a good estimate for $\sum_{i}^{N}(Y_i - WX_i)^2$. When $M$ is very small, they can be very different from each other.

$\endgroup$

1 Answer 1

4
$\begingroup$

The loss function is divided by N to ensure that it does not depend on the number of training observations allowing for better comparision across models.

Yes, there are good enough reasons to divide by the mini-batch size while updating the loss function . In batch gradient descent the loss divided by batch-size introduced to make the cost function comparable across different size datasets gets automatically applied to the weight decay term.

Also in terms of regularisation, when size of the training data decreases it's representativeness of overall distribution reduces, which automatically scales up the regularisation. When M<N , dividing by M will have more regularisation effect than N which is desirable.

$\endgroup$
1
  • $\begingroup$ Also, the mini-batch gradient does not try to get a very close estimation of full gradient. Instead it tries to capture the direction of the gradient, so that you still move in the correct direction using the mini batch gradient. $\endgroup$
    – Victor Luu
    Jul 29, 2020 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.