2
$\begingroup$

In Neural Network examples that I have seen online - sometimes the Mean Square Error is presented as

$$ MSE = \frac{{1}}{2n} \sum_{i}^{n} ( \widehat{y_i} -y_i)^2 \quad (1) $$

and other times

$$ MSE = \frac{{1}}{2} \sum_{i}^{n} ( \widehat{y_i} -y_i)^2 \quad (2) $$

Where I guess $n$ is the number of output nodes.

Which one is the correct formula to use? Are we minimizing the total error or the "average" error between each output as in linear regression?

Also, bonus question: Do we really need to multiply by $1/2$? In my opinion, it is not that much more convenient than multiplying by $2$ when we take the derivative.

Thank you

Update based on feedback below

$$ MSE = \frac{{1}}{n} \sum_{i}^{n} (y_i - \widehat{y_i})^2 \quad (3) $$

Where $y_i$ is the desired Neural Network output, and $\widehat{y_i}$ is the neural network output.

$\endgroup$
  • 3
    $\begingroup$ If there isn't an $n^{-1}$ out the front (whether combined with any other constant or not), in what sense would a sum of squares be considered a mean? If it's really just a sum, why would they not just call it that? (I find such practices - not so uncommon in machine learning - to be frankly bizarre. It's as if the authors responsible were actively trying to be confusing.) $\endgroup$ – Glen_b Mar 16 at 23:55
  • 2
    $\begingroup$ @Glen_b I am in violent agreement with you. $\endgroup$ – Edv Beq Mar 17 at 0:00
1
$\begingroup$

What matters is the sum of squares. The rest is not important mathematically for optimization of a finite sample. Obviously, for an infinite size sample the sum is infinite, and things don't work out so well.

$\endgroup$
  • $\begingroup$ So whether I use $1/2$ or $1/2n$ I get the same results? I am speaking in the context of Neural Networks $\endgroup$ – Edv Beq Nov 10 '17 at 22:45
  • $\begingroup$ In NN you'll never get exactly the same answers because of use of stochastic optimization procedures; but yes the results should be very close $\endgroup$ – Aksakal Nov 10 '17 at 23:07
  • 2
    $\begingroup$ but you will likely need to adjust the learning rate appropriately when switching- taking into account the scaling of the objective function $\endgroup$ – seanv507 Nov 11 '17 at 7:03
  • $\begingroup$ @seanv507, yes, when math is translated into software you have to consider what's lost in translation, things like precision, rounding etc. will bring the differences between otherwise mathematically identical approaches. in ideal world the learning rate would not matter, after all you'll find the solution eventually; in real it does matter a lot both in terms of computational time and in terms of accumulation of rounding errors $\endgroup$ – Aksakal Nov 11 '17 at 15:10
  • $\begingroup$ I find your second sentence misleading. While getting the mean or not is not important for optimization mathematically, it certainly can make a difference for optimization algorithms. Not in terms of rounding errors, but in terms of the scale of the gradient of MSE. $\endgroup$ – Soroush Mar 28 at 22:51
4
$\begingroup$

It's perfectly straightforward, I think:

To get mean square error, you take the errors ($y_i-\hat{y}_i$ -- and no, the error is not $\hat{y}_i-y_i$), you square them and then take their mean. Mean-square-error, just like it says on the label.

So, correctly, $MSE = \frac{{1}}{n} \sum_{i}^{n} ( y_i-\hat{y_i})^2$

(Anything else will be some other object)

If you don't divide by $n$, it can't really be called a mean; without $\frac{1}{n}$, that's a sum not a mean.

The additional factor of $\frac12$ means that it isn't MSE either, but half of MSE.

That said, you can halve it or double it, multiply by $n$ and so on without changing the argmin so if that's all you're doing that won't matter -- but a statistician does more with MSE than merely minimize it, so to me it matters to call things by their right names.

If - and here I address the authors you're reading, not you the innocent asker - you want to halve MSE, call it half-MSE. If you calculate SSE, say so. Don't play silly games with perfectly descriptive terms because you can't be bothered to type 5 extra characters. It's not hard!

$\endgroup$
  • $\begingroup$ I agree - those are very good points. $\endgroup$ – Edv Beq Nov 11 '17 at 13:48
  • $\begingroup$ I never defined the MSE terms but please take a look at my update. $\endgroup$ – Edv Beq Nov 11 '17 at 14:10
  • $\begingroup$ @edv yes, I realize the definitions you quoted were someone else's. It's simply a matter of terminology, but in response to "which of those two things is MSE" (a question about terminology) I really wanted to say "neither" and explain why -- and to complain to the people who perpetuate such muddying of the water. $\endgroup$ – Glen_b Nov 11 '17 at 21:19
0
$\begingroup$

You're seeing both equations because they're used at different points in training.

$$\frac{{1}}{2} \sum_{i}^{n} ( y_i - \widehat{y_i})^2 \quad$$

is the error of a single training example, where n is the number of output nodes. The second equation,

$$\frac{{1}}{2n} \sum_{i}^{n} ( y_i - \widehat{y_i})^2 \quad$$

is the error of the network across all n training examples, though these are usually broken up into mini-batches to make calculation less expensive.

Also, $(\widehat{y_i} -y_i)^2 = (y_i - \widehat{y_i})^2$.

Source: http://neuralnetworksanddeeplearning.com/chap2.html

Old question but still the top Google result for "MSE neural network"

$\endgroup$
  • $\begingroup$ I still don't get it. The first example doesn't look like a mean of anything, as the original commenters pointed out. And the linked source refers to these as "costs", not MSE. $\endgroup$ – The Laconic Mar 17 at 0:33
0
$\begingroup$

Each sample in your minibatch gives you a squared error $(y_i-\hat{y_i})^2$. $MSE$ is the mean of this value for all $n$ samples in your minibatch: $$MSE=\frac{1}{n}\sum_{i=1}^{n}(\hat{y_i}-y_i)^2$$ Notes

  • There is no $\frac{1}{2}$ factor in $MSE$.
  • Both $MSE$ and $(\hat{y_i}-y_i)^2$ are scalars.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.