3
$\begingroup$

I saw an example on a blog where a k nearest neighbors algorithm was run to predict responses on a test data set.The MSE(mean squared error) of the predictions was larger than the variance of the response variable in the test set. Shouldn't the MSE always be less than the variance of the response variable as SSTO(total sum of squares) = RSS(model sum of squares) + SSE(error sum of squares).


$\endgroup$
5
  • 2
    $\begingroup$ Your question is full of acronyms, please state what's MSE, SSTO, RSS, and SSE, so everyone can understand it, and perhaps it might increase its visibility. $\endgroup$
    – Firebug
    Dec 13, 2016 at 17:07
  • 1
    $\begingroup$ To the question, I don't even see the reason why test mean-squared-error would have to be lesser than training data response variance. $\endgroup$
    – Firebug
    Dec 13, 2016 at 17:18
  • $\begingroup$ @Firebug I am comparing the test mean square error with test data response variance $\endgroup$
    – zorny
    Dec 13, 2016 at 17:21
  • 1
    $\begingroup$ Okay, but it doesn't change anything. I still can't grasp the premise of this question, why would test MSE have to be smaller than variance? $\endgroup$
    – Firebug
    Dec 13, 2016 at 18:50
  • 2
    $\begingroup$ In linear regression this makes sense on training data only. $\endgroup$
    – Firebug
    Dec 13, 2016 at 18:51

2 Answers 2

2
$\begingroup$

Case 1:

SST (total) = RSS (model) + SSE (error)

This is only true in general if the model includes an intercept (and, in relation to case 3, if the mean of the errors is 0).

(more precisely, you do not need an explicit intercept, for instance for linear regression it is sufficient that the intercept is in the column space of the regressors)

See for instance the following y=bx fit:

demonstrating exception to SST = RSS + SSE

Clearly the error is much larger than the total variance in the data.


Case 2:

Other issues may be when you are comparing training vs test, but it is unclear whether this is the case in this question. (there is an explicit 'in the test set' mentioned, but is this the same for all terms?)

(see for instance https://stats.stackexchange.com/a/299523/164061 and the four other links in that post)

Case 3:

In the test set a similar effect may occur as with the lack of the intercept. If the model (which you fitted to the training set, and not to the test set) is not a good fit for the test set, then you may get a larger error than the total variance. (this is what Firebug meant with, 'makes sense to the training data only)

This bad fitting, for which the SSE becomes larger than the SST, is especially likely when you are over-fitting.

Case 4:

If you are fitting a Bayesian model, or some model that has bias (e.g. regularisation) then there is no guarantee that SST (total) > RSS (model).

$\endgroup$
1
$\begingroup$

The MSE is defined as $MSE = N^{-1} \sum_{i=1}^N (y_i - \hat{y}_i)^2$, where $\hat{y}_i$ is the fitted, or predicted, value.

The variance is $V = N^{-1} \sum_{i=1}^N (y_i - \bar{y})^2$, where $ \bar{y} = \sum_{i=1}^N y_i$.

It is true that in most cases, MSE will tend to be smaller than $V$, but it depends inherently on how the model fits the values $\hat{y}_i$ to the data. Given that there is some noise and some signal in the data, it will often be the case that the MSE is smaller, but not always.

$\endgroup$
6
  • $\begingroup$ I understand that the MSR and MSE and be different but since the MSR + MSE = MST shouldn't the variance always be larger? $\endgroup$
    – zorny
    Dec 13, 2016 at 16:56
  • $\begingroup$ I totally see where the confusion is coming form and I agree with you. But what are MSR and MST? I guess that one of those must be a definition term (the remainder), which must therefore be what can become negative. Perhaps your intuition comes from the linear regression case? $\endgroup$ Dec 13, 2016 at 17:03
  • $\begingroup$ @Superproner. Yes my intuition comes from linear regression. Please see this link: onlinecourses.science.psu.edu/stat501/node/266. MSR is "regression mean square (MSR)" and MST is the mean total sum of squares. Isn't this applicable for any model. Total variance = explained variance + unexplained variance so unexplained variance cannot be greater than total variance $\endgroup$
    – zorny
    Dec 13, 2016 at 17:17
  • $\begingroup$ I believe that for the linear regression, it is a result (as you say). But for other models, it need not hold. $\endgroup$ Dec 13, 2016 at 18:15
  • $\begingroup$ Is there an intuitive explanation for this? $\endgroup$
    – zorny
    Dec 13, 2016 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.