6
$\begingroup$

I was studying the first chapter of Pattern Recognition and Machine Learning, by Christopher Bishop, and in the presentation of the sum of squared errors function

$$ E(w) = \frac{1}{2}\ \sum\limits_{n=1}^N \{y(x_n,w) - t_n\}^2 $$

where $y$ is the polinomial function being modelled, $x$ its variable, $w$ the polinomial coefficients to be discovered and $N$ the size of the training set.

In this book, and in other situations such as the Andrew Ng's video lectures, this $ \frac{1}{2} $ is "included for later convenience". Which convenience is that? I'm struggling to ignore this but, I simply can't avoid the fact that I don't know where this fraction is comming from and what is its impact on this calculation.

What this one half means in the context of the sum of squared errors?

$\endgroup$
8
  • 5
    $\begingroup$ What happens to the $1/2$ when you differentiate $E$? $\endgroup$
    – whuber
    Commented Jun 17, 2015 at 15:08
  • $\begingroup$ It goes away, but which mathematical principle gives me the "permission" to freely add this 1/2 just for differentiation? $\endgroup$
    – chicao
    Commented Jun 17, 2015 at 15:10
  • 5
    $\begingroup$ Consider what the sum of squared errors is being used for. If you care about it in and of itself, you would be cautious about multiplying it by $1/2$. But if you are trying to find values of $w$ that minimize it, then rescaling it by any nonzero number will not change the answer. $\endgroup$
    – whuber
    Commented Jun 17, 2015 at 15:17
  • 3
    $\begingroup$ Tiny nitpickery on @whuber's comment - this works for rescaling by any positive number. If you rescale by a negative number, then you need to switch from minimization to maximization. Like in this case, going from minimizing the sums of squared errors to maximizing the likelihood. $\endgroup$ Commented Jun 17, 2015 at 15:25
  • 3
    $\begingroup$ Thank you all for the kind answers. It appears more clear to me now regarding the minimization problem, since minimizing a function and half a function does not effect the fact that you are minimizing. Thank you all again. $\endgroup$
    – chicao
    Commented Jun 17, 2015 at 18:50

2 Answers 2

5
$\begingroup$

As long as $E(w) \geq 0$ (which is true for this sum of squares), minimizing $(1/2)E(w)$ is equivalent to minimizing $E(w)$. As has been pointed out in the comments, the factor of $1/2$ disappears when you take the derivative of $E(w)$.

$\endgroup$
4
  • $\begingroup$ Contrasted with the appearance of a factor of $2$ if you leave it out, complicating downstream computations. $\endgroup$ Commented Jun 17, 2015 at 16:49
  • $\begingroup$ Hi @Brain. I understand your answer. Thank you very much. Could you please edit you latex formatting ? In my browser is everything overlapped. $\endgroup$
    – chicao
    Commented Jun 17, 2015 at 18:51
  • $\begingroup$ You might have a browser problem. There is nothing problematic about the $\TeX$ markup here. (I get this overlapping on an Android phone.) $\endgroup$
    – whuber
    Commented Jun 17, 2015 at 19:27
  • $\begingroup$ I thought that would be related to the maximum likelihood of the normal distribution. $\endgroup$ Commented Nov 16, 2016 at 20:54
1
$\begingroup$

It probably doesn't matter whether you use $\frac{1}{2}$ or $\frac{1}{n}$ for MSE because the denominator value of 2 and $n$ will never change for the dataset being evaluated. The scale of both methods will differ due to the magnitude of what's calculated, but nevertheless, you'll be dividing by a constant that never changes. If you compare MSE across datasets, then you might go with $\frac{1}{n}$, since that will scale with sample size -- however, within the algorithm being fitted, the artificial neural network (ANN) just needs a reference point to gauge how bad/good the fit is.

FYI-The same equation (i.e., $\frac{1}{2}$), is used for MSE in the neural network chapter of Friedman and Tibshirani (Statistical Learning, Springer). Recall, however, MSE is for continuous function approximation using an ANN and cross-entropy is used for classification problems for ANNs.

Since you're reading Bishop, what you won't pull away from reading is that a key issue with ANNs is that they like input features to have range [-1,1] with no correlation between features. If there is correlation between features, then an ANN will spend time learning the correlation -- which is what you don't want an algorithm to do. Therefore, run PCA to decorrelate the features first and then input the top 10 orthogonal PC's into the ANN.

Last, there is another primary issue with ANNs regarding input samples, which is related to redundancy. That is, many of your records may be the same, and inputting the same (similar) records into an ANN does not help. One of the only groups I know who have developed methods to collapse features and redundant samples simultaneously before input to an ANN is Jurik Research's (DDR)

Finally, look at Ripley's text on ANNs, since the primary focus will always be how you regularized in order to minimize over-fitting and maximize generalization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.