0
$\begingroup$

In common statistical textbooks' linear regression topic, Mean Squared Error is often defined as $$MSE = \dfrac{(y-\hat{y})^T(y-\hat{y})}{n-p} = \dfrac{RSS}{n-p}$$ where the $y$ and $\hat{y}$ is a column vector containing each y and its predicted values respectively, RSS is the residual sum of squares, and n is the number of training samples, p being the number of variables including the intercept term. It can be shown that this definition of MSE is the unbiased estimator of the variance of the error term $\varepsilon$.

Now, the question comes, in Python's sklearn, or in fact in most ML books, MSE is just simply defined as the more intuitive (at least for me) $$MSE = \dfrac{RSS}{n}$$

Although one can argue as $n \to \infty$, both MSE converges to the same values. But I still get very confused over which one to use and why the difference? If someone can link me or explain to me it would be great; I tried searching but I dont see a common answer.

Improve this question
$\endgroup$
1
  • $\begingroup$ Two points worth mentioning are 1) when you optimize the loss function to get parameter estimates, both denominators result in the same value that minimizes the loss (as does straight residual sum of squares) and 2) if you want to compare models to see which is performing better, as long as you use the same denominator for evaluating each model, it doesn't matter what denominator you use (again, RSS is adequate for this task, too). $\endgroup$
    – Dave
    Feb 6, 2020 at 18:57

1 Answer 1

Reset to default
4
$\begingroup$

The formula that divides by $n-p$ is an estimator for unbiased mean squared error for regression, where you divide by the degrees of freedom. Imagine you would like to compare results returned by different machine learning algorithms, in many cases it would be hard to say what does exactly $p$ is, and for some models, like non-parametric models, or neural networks, $n-p$ could go all the way down to zero, or get negative, so this wouldn't have much sense. That is why in machine learning we use the simplified version, that works no matter what algorithms do you compare with each other.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, seems I have a long way to go to understand the difference between ML and Statistics. So in general, the correct MSE is the one with n-p as the denominator, but in order to compare different models, we can be less pedantic and drop the p. $\endgroup$
    – nan
    Feb 5, 2020 at 15:01
  • 2
    $\begingroup$ @nan it depends what you mean by "correct". In ML you focus on train/validation/test performance, dividing by $n$ is fine for this purpose. If you want to read about the differences between stats vs ML, check stats.stackexchange.com/questions/6/… $\endgroup$
    – Tim ♦
    Feb 5, 2020 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.