4
$\begingroup$

In Neural Network examples that I have seen online - sometimes the Mean Square Error is presented as

$$ MSE = \frac{{1}}{2n} \sum_{i}^{n} ( \widehat{y_i} -y_i)^2 \quad (1) $$

and other times

$$ MSE = \frac{{1}}{2} \sum_{i}^{n} ( \widehat{y_i} -y_i)^2 \quad (2) $$

Where I guess $n$ is the number of output nodes.

Which one is the correct formula to use? Are we minimizing the total error or the "average" error between each output as in linear regression?

Also, bonus question: Do we really need to multiply by $1/2$? In my opinion, it is not that much more convenient than multiplying by $2$ when we take the derivative.

Thank you

Update based on feedback below

$$ MSE = \frac{{1}}{n} \sum_{i}^{n} (y_i - \widehat{y_i})^2 \quad (3) $$

Where $y_i$ is the desired Neural Network output, and $\widehat{y_i}$ is the neural network output.

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ If there isn't an $n^{-1}$ out the front (whether combined with any other constant or not), in what sense would a sum of squares be considered a mean? If it's really just a sum, why would they not just call it that? (I find such practices - not so uncommon in machine learning - to be frankly bizarre. It's as if the authors responsible were actively trying to be confusing.) $\endgroup$
    – Glen_b
    Mar 16, 2019 at 23:55
  • 3
    $\begingroup$ @Glen_b I am in violent agreement with you. $\endgroup$
    – Edv Beq
    Mar 17, 2019 at 0:00
  • $\begingroup$ Is n the number of output nodes or the number of samples? I am confused... $\endgroup$
    – Funkwecker
    Aug 8, 2020 at 15:31

6 Answers 6

Reset to default
2
$\begingroup$

What matters is the sum of squares. The rest is not important mathematically for optimization of a finite sample. Obviously, for an infinite size sample the sum is infinite, and things don't work out so well.

Improve this answer
$\endgroup$
5
  • $\begingroup$ So whether I use $1/2$ or $1/2n$ I get the same results? I am speaking in the context of Neural Networks $\endgroup$
    – Edv Beq
    Nov 10, 2017 at 22:45
  • $\begingroup$ In NN you'll never get exactly the same answers because of use of stochastic optimization procedures; but yes the results should be very close $\endgroup$
    – Aksakal
    Nov 10, 2017 at 23:07
  • 3
    $\begingroup$ but you will likely need to adjust the learning rate appropriately when switching- taking into account the scaling of the objective function $\endgroup$
    – seanv507
    Nov 11, 2017 at 7:03
  • 1
    $\begingroup$ @seanv507, yes, when math is translated into software you have to consider what's lost in translation, things like precision, rounding etc. will bring the differences between otherwise mathematically identical approaches. in ideal world the learning rate would not matter, after all you'll find the solution eventually; in real it does matter a lot both in terms of computational time and in terms of accumulation of rounding errors $\endgroup$
    – Aksakal
    Nov 11, 2017 at 15:10
  • $\begingroup$ I find your second sentence misleading. While getting the mean or not is not important for optimization mathematically, it certainly can make a difference for optimization algorithms. Not in terms of rounding errors, but in terms of the scale of the gradient of MSE. $\endgroup$
    – Soroush
    Mar 28, 2019 at 22:51
8
$\begingroup$

It's perfectly straightforward, I think:

To get mean square error, you take the errors ($y_i-\hat{y}_i$ -- and no, the error is not $\hat{y}_i-y_i$), you square them and then take their mean. Mean-square-error, just like it says on the label.

So, correctly, $MSE = \frac{{1}}{n} \sum_{i}^{n} ( y_i-\hat{y_i})^2$

(Anything else will be some other object)

If you don't divide by $n$, it can't really be called a mean; without $\frac{1}{n}$, that's a sum not a mean.

The additional factor of $\frac12$ means that it isn't MSE either, but half of MSE.

That said, you can halve it or double it, multiply by $n$ and so on without changing the argmin so if that's all you're doing that won't matter -- but a statistician does more with MSE than merely minimize it, so to me it matters to call things by their right names.

If - and here I address the authors you're reading, not you the innocent asker - you want to halve MSE, call it half-MSE. If you calculate SSE, say so. Don't play silly games with perfectly descriptive terms because you can't be bothered to type 5 extra characters. It's not hard!

Improve this answer
$\endgroup$
3
  • $\begingroup$ I agree - those are very good points. $\endgroup$
    – Edv Beq
    Nov 11, 2017 at 13:48
  • $\begingroup$ I never defined the MSE terms but please take a look at my update. $\endgroup$
    – Edv Beq
    Nov 11, 2017 at 14:10
  • 1
    $\begingroup$ @edv yes, I realize the definitions you quoted were someone else's. It's simply a matter of terminology, but in response to "which of those two things is MSE" (a question about terminology) I really wanted to say "neither" and explain why -- and to complain to the people who perpetuate such muddying of the water. $\endgroup$
    – Glen_b
    Nov 11, 2017 at 21:19
1
$\begingroup$

You're seeing both equations because they're used at different points in training.

$$\frac{{1}}{2} \sum_{i}^{n} ( y_i - \widehat{y_i})^2 \quad$$

is the error of a single training example, where n is the number of output nodes. The second equation,

$$\frac{{1}}{2n} \sum_{i}^{n} ( y_i - \widehat{y_i})^2 \quad$$

is the error of the network across all n training examples, though these are usually broken up into mini-batches to make calculation less expensive.

Also, $(\widehat{y_i} -y_i)^2 = (y_i - \widehat{y_i})^2$.

Source: http://neuralnetworksanddeeplearning.com/chap2.html

Old question but still the top Google result for "MSE neural network"

Improve this answer
$\endgroup$
3
  • $\begingroup$ I still don't get it. The first example doesn't look like a mean of anything, as the original commenters pointed out. And the linked source refers to these as "costs", not MSE. $\endgroup$
    – The Laconic
    Mar 17, 2019 at 0:33
  • $\begingroup$ That author does cal it both a cost function and MSE: "We'll call C the quadratic cost function; it's also sometimes known as the mean squared error or just MSE." However, we can squarely say he's incorrect to call it MSE without clarifying the extra use of the 2. The error from one training sample will not have a 2, not will the squared error. He is, however, free to define his own cost function. Calling it the MSE, it the mistake. neuralnetworksanddeeplearning.com/chap1.html $\endgroup$
    – VISQL
    Sep 24, 2019 at 14:23
  • $\begingroup$ I feel this should be the correct answer $\endgroup$
    – Paul Matthews
    Jun 19, 2021 at 5:27
1
$\begingroup$

Each sample in your minibatch gives you a squared error $(y_i-\hat{y_i})^2$. $MSE$ is the mean of this value for all $n$ samples in your minibatch: $$MSE=\frac{1}{n}\sum_{i=1}^{n}(\hat{y_i}-y_i)^2$$ Notes

  • There is no $\frac{1}{2}$ factor in $MSE$.
  • Both $MSE$ and $(\hat{y_i}-y_i)^2$ are scalars.
Improve this answer
$\endgroup$
0
$\begingroup$

The factor of two is used to cancel the exponent 2 at every differentiation. Otherwise, MSE is defined as in the answer by Soroush.

Improve this answer
$\endgroup$
0
$\begingroup$

I see that: because there are three types of layers (input, Hidden, and output) then dividing by (2) is correct, that's the number (n) is belongs the number of layers minuses one

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.