6
$\begingroup$

I was studying the first chapter of Pattern Recognition and Machine Learning, by Christopher Bishop, and in the presentation of the sum of squared errors function

$$ E(w) = \frac{1}{2}\ \sum\limits_{n=1}^N \{y(x_n,w) - t_n\}^2 $$

where $y$ is the polinomial function being modelled, $x$ its variable, $w$ the polinomial coefficients to be discovered and $N$ the size of the training set.

In this book, and in other situations such as the Andrew Ng's video lectures, this $ \frac{1}{2} $ is "included for later convenience". Which convenience is that? I'm struggling to ignore this but, I simply can't avoid the fact that I don't know where this fraction is comming from and what is its impact on this calculation.

What this one half means in the context of the sum of squared errors?

$\endgroup$
  • 5
    $\begingroup$ What happens to the $1/2$ when you differentiate $E$? $\endgroup$ – whuber Jun 17 '15 at 15:08
  • $\begingroup$ It goes away, but which mathematical principle gives me the "permission" to freely add this 1/2 just for differentiation? $\endgroup$ – chicao Jun 17 '15 at 15:10
  • 5
    $\begingroup$ Consider what the sum of squared errors is being used for. If you care about it in and of itself, you would be cautious about multiplying it by $1/2$. But if you are trying to find values of $w$ that minimize it, then rescaling it by any nonzero number will not change the answer. $\endgroup$ – whuber Jun 17 '15 at 15:17
  • 3
    $\begingroup$ Tiny nitpickery on @whuber's comment - this works for rescaling by any positive number. If you rescale by a negative number, then you need to switch from minimization to maximization. Like in this case, going from minimizing the sums of squared errors to maximizing the likelihood. $\endgroup$ – Stephan Kolassa Jun 17 '15 at 15:25
  • 3
    $\begingroup$ Thank you all for the kind answers. It appears more clear to me now regarding the minimization problem, since minimizing a function and half a function does not effect the fact that you are minimizing. Thank you all again. $\endgroup$ – chicao Jun 17 '15 at 18:50
4
$\begingroup$

As long as $E(w) \geq 0$ (which is true for this sum of squares), minimizing $(1/2)E(w)$ is equivalent to minimizing $E(w)$. As has been pointed out in the comments, the factor of $1/2$ disappears when you take the derivative of $E(w)$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Contrasted with the appearance of a factor of $2$ if you leave it out, complicating downstream computations. $\endgroup$ – Matthew Drury Jun 17 '15 at 16:49
  • $\begingroup$ Hi @Brain. I understand your answer. Thank you very much. Could you please edit you latex formatting ? In my browser is everything overlapped. $\endgroup$ – chicao Jun 17 '15 at 18:51
  • $\begingroup$ You might have a browser problem. There is nothing problematic about the $\TeX$ markup here. (I get this overlapping on an Android phone.) $\endgroup$ – whuber Jun 17 '15 at 19:27
  • $\begingroup$ I thought that would be related to the maximum likelihood of the normal distribution. $\endgroup$ – Lerner Zhang Nov 16 '16 at 20:54
1
$\begingroup$

It probably doesn't matter whether you use $\frac{1}{2}$ or $\frac{1}{n}$ for MSE because the denominator value of 2 and $n$ will never change for the dataset being evaluated. The scale of both methods will differ due to the magnitude of what's calculated, but nevertheless, you'll be dividing by a constant that never changes. If you compare MSE across datasets, then you might go with $\frac{1}{n}$, since that will scale with sample size -- however, within the algorithm being fitted, the artificial neural network (ANN) just needs a reference point to gauge how bad/good the fit is.

FYI-The same equation (i.e., $\frac{1}{2}$), is used for MSE in the neural network chapter of Friedman and Tibshirani (Statistical Learning, Springer). Recall, however, MSE is for continuous function approximation using an ANN and cross-entropy is used for classification problems for ANNs.

Since you're reading Bishop, what you won't pull away from reading is that a key issue with ANNs is that they like input features to have range [-1,1] with no correlation between features. If there is correlation between features, then an ANN will spend time learning the correlation -- which is what you don't want an algorithm to do. Therefore, run PCA to decorrelate the features first and then input the top 10 orthogonal PC's into the ANN.

Last, there is another primary issue with ANNs regarding input samples, which is related to redundancy. That is, many of your records may be the same, and inputting the same (similar) records into an ANN does not help. One of the only groups I know who have developed methods to collapse features and redundant samples simultaneously before input to an ANN is Jurik Research's (DDR)

Finally, look at Ripley's text on ANNs, since the primary focus will always be how you regularized in order to minimize over-fitting and maximize generalization.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.