Why is it necessary to divide by the number of samples when optimizing squared error? In a lot of different optimization problems, and with particular regard to gradient descent, we use the mean squared error as a loss function. In the formulation of mean squared error, you divide by the number of samples - that is, we use the expression $ \frac{1}{n} \sum\limits_n (y_i - g(x_i))^2$ where $g(x)$ is some parametrized estimating function. Because we are performing a minimization over this function, it seems like the $\frac{1}{n}$ term is unnecessary, and minimizing with respect to each of the parameters should obtain the same result. Is it important to multiply the sum squared error by $\frac{1}{n}$, and if so, why? (I understand that it is true to the idea of taking some "mean" value, but I'm not clear on if there is some theoretic necessity for this, or if it is just a standard formulation, Apologies in advance if I'm missing something blatantly obvious).
 A: If you are truly only optimizing mean squared error, it is true that multiplying the objective by $n$ will give you the same solution.
Mean squared error is often preferred over sum squared error, though, because:

*

*It's typically easier to interpret.

*It's easier to compare values if you're doing cross-validation or something else where you might end up with different-sized training sets.

*If you're regularizing the objective in any way, say $\frac{1}{n} \sum_{i=1}^n (y_i - g(x_i))^2 + \lambda P(g)$, then it might "make more sense" to use a consistent regularization value for different-sized training sets. (Typically, you do want $\lambda$ to scale with $n$; the theoretically optimal choice for ridge regression is usually $\frac{1}{\sqrt n}$ in the normalized case or $\sqrt n$ in the unnormalized one – but I find it easier to think of "you need less regularization for larger $n$" in the normalized case than in the unnormalized one.)

*The same kind of considerations hold for an optimizer's step size as for the regularization weight, as pointed out in the comments.

But, as long as you keep these kinds of considerations in mind, you can absolutely drop the $\frac1n$ factor; it doesn't really change anything.
