2
$\begingroup$

I was reading about the probabilistic interpretation of linear regression and the following formula is derived using maximum likelihood estimates : $$ \begin{align*} β=\underset{β}{\text{argmin}}\sum_{i=1}^{n}(y_i−(β_0+β_1x_1+...+β_px_p))^2 =\underset{β}{\text{argmin}}\sum_{i=1}^{n}(y_i−\hat{y_i})^2 \end{align*} $$

where $β$ is the weight vector , $y$ is the target variable and $\hat{y}$ is the hypothesis.

The only thing I am not getting is the $1/N$ (where $N$ is the number of samples) term . The above equation is just the sum of squared error. But we always use mean squared error. So, is there any mathematical proof of the same ?

$\endgroup$
4
  • 2
    $\begingroup$ For a given dataset with a fixed sample size, whatever minimises the sum of squared errors also minimises the mean of those squared errors. For example if the data are 1, 2, 3 the sum of squared errors is minimised by 2, which is the mean; you'd get the same answer if you minimised the mean squared error; the two have the same minimand. The principle carries over to estimating two or more parameters in a regression. (If you plot the function as a quadratic, dividing by sample size just changes the axis labels; it doesn't move the function.) $\endgroup$
    – Nick Cox
    Commented Oct 11, 2020 at 8:41
  • $\begingroup$ @Nick But then , is there any reason behind taking mean of errors ? $\endgroup$ Commented Oct 11, 2020 at 10:51
  • $\begingroup$ The size of the mean can be interpreted in terms of the size of the original observations. $\endgroup$ Commented Oct 11, 2020 at 10:54
  • $\begingroup$ The mean squared error is a step towards its square root, which can be useful as a standard error. In practice people often use a different denominator for other reasons. This is discussed in detail in just about any regression text. $\endgroup$
    – Nick Cox
    Commented Oct 11, 2020 at 10:54

1 Answer 1

1
$\begingroup$

It is a fact from calculus that $\arg\min$ does not change when we apply an increasing function like multiplication by a positive number. For instance, $f(x)=x^2$ and $f(x)=7x^2$ have the same minimizer, $x=0$.

Consequently, the sum of the squares errors (technically residuals) and the mean of the squares residuals have extremely similar behavior: whatever regression parameters or predictions minimizes one also minimizes the other.

The reason why people would divide by the sample size is to keep a large sample size from giving an enormous number. The typical squared residual could be quite small, but the sum of a million or billion small numbers might wind up being quite large. Additionally, there is a relationship between the mean squared error (again, technically residuals) and the variance of the residuals in that the mean squared error formula is exactly the “population” variance of the residuals.

For instance, imagine your boss asking you about model performance. Wouldn’t you prefer to report, and your boss find more useful, information about how large a typical residual is, rather than the sum of all of the squared residuals?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.