2
$\begingroup$

In a lot of different optimization problems, and with particular regard to gradient descent, we use the mean squared error as a loss function. In the formulation of mean squared error, you divide by the number of samples - that is, we use the expression $ \frac{1}{n} \sum\limits_n (y_i - g(x_i))^2$ where $g(x)$ is some parametrized estimating function. Because we are performing a minimization over this function, it seems like the $\frac{1}{n}$ term is unnecessary, and minimizing with respect to each of the parameters should obtain the same result. Is it important to multiply the sum squared error by $\frac{1}{n}$, and if so, why? (I understand that it is true to the idea of taking some "mean" value, but I'm not clear on if there is some theoretic necessity for this, or if it is just a standard formulation, Apologies in advance if I'm missing something blatantly obvious).

$\endgroup$

1 Answer 1

8
$\begingroup$

If you are truly only optimizing mean squared error, it is true that multiplying the objective by $n$ will give you the same solution.

Mean squared error is often preferred over sum squared error, though, because:

  • It's typically easier to interpret.
  • It's easier to compare values if you're doing cross-validation or something else where you might end up with different-sized training sets.
  • If you're regularizing the objective in any way, say $\frac{1}{n} \sum_{i=1}^n (y_i - g(x_i))^2 + \lambda P(g)$, then it might "make more sense" to use a consistent regularization value for different-sized training sets. (Typically, you do want $\lambda$ to scale with $n$; the theoretically optimal choice for ridge regression is usually $\frac{1}{\sqrt n}$ in the normalized case or $\sqrt n$ in the unnormalized one – but I find it easier to think of "you need less regularization for larger $n$" in the normalized case than in the unnormalized one.)
  • The same kind of considerations hold for an optimizer's step size as for the regularization weight, as pointed out in the comments.

But, as long as you keep these kinds of considerations in mind, you can absolutely drop the $\frac1n$ factor; it doesn't really change anything.

$\endgroup$
5
  • $\begingroup$ Could we also say that normalizing is needed in gradient descent algorithms? I mean in the first case (unormalized MSE) we update the weights as: $$w \gets w - \eta \nabla \mathcal{L}$$ while in the second case (normalized MSE) we update the weights as: $$ w \gets w - \eta \frac{1}{n} \nabla \mathcal{L}$$ That is, if we don't normalize $\mathcal{L}$ we may "jump" around the minimum. $\endgroup$
    – ado sar
    Feb 16, 2023 at 17:21
  • 1
    $\begingroup$ @adosar Remember that $\eta$ is just an arbitrary parameter; the two are exactly the same if you scale $\eta$ by $n$. But if you change $n$, then one value of $\eta$ has a "more similar meaning" in the MSE case than in the SSE case. $\endgroup$
    – Danica
    Feb 24, 2023 at 19:19
  • $\begingroup$ The factor $\frac{1}{n}$ takes into account only the squared error. Should the equation of the loss be modified when we train models with many parameters (e.g. neural nets which can have million of weights)? That is, should the loss be: $$ \mathcal{L} = \frac{1}{n} MSE + λ\mathcal{P}(g)$$ or $$ \mathcal{L} = \frac{1}{n} \left[MSE + λ\mathcal{P}(g)\right]$$ ? $\endgroup$
    – ado sar
    May 25, 2023 at 21:54
  • $\begingroup$ @adosar I think everything I said in the answer applies to the deep case as well as the linear case. Again, it doesn’t really mean anything, it just scales your arbitrary value of $\lambda$. $\endgroup$
    – Danica
    May 26, 2023 at 7:45
  • $\begingroup$ I see. From an optimization point of view, the optimal solution still can be found (although a different $\lambda$ will be required, the scale factor you mentioned). +1 for the 3d bullet (the $\frac{1}{n}$ factor makes more sense only for the $MSE$, so $\lambda$ doesn't scale with $n$). $\endgroup$
    – ado sar
    May 26, 2023 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.