3
$\begingroup$

For a given regression dataset, with target $Y \in \mathbb{R}$, where the training set examples have $\mathrm{std}(Y)$ and $\mathrm{mean}(Y)$, what is a mean squared error that makes the result obvious or not interesting? What would be a "reasonable MSE"? (all in relation to the standard deviation and the mean of $Y$).

For example, it doesn't make much sense for such MSE to be larger than the standard deviation squared (variance), because choosing a constant target to be the mean would give such error?

$\endgroup$
2
  • 1
    $\begingroup$ Well, one option would be to, as you figured out, to use a constant - such as mean - as a simple forecast, and compare it to that. This is what $R^2$ does, compares your model to a constant forecast model (plus another constant). $\endgroup$ Jun 23 at 12:31
  • $\begingroup$ Related, though not identical: stats.stackexchange.com/questions/414349/… $\endgroup$
    – mkt
    Jun 23 at 12:33

2 Answers 2

4
$\begingroup$

You partially answered yourself. The best simple (single parameter) model that you can have to minimize squared error is to always predict the mean. In such a case, the mean squared error would be equal to the variance. What follows, if your solution is worse than this, it's really bad. Other than that, MSE or RMSE are not bounded, they can take any non-negative values. The only way to judge if the value of the metric is reasonable is by comparing it to some benchmark, like the results of another model.

$\endgroup$
5
$\begingroup$

You would want the MSE to be as small as possible compared to the variance. A standard measure for this is the coefficient of determination, also known as "R squared": $$ R^2 = 1 - \frac{MSE}{var(Y)}. $$ But what is "reasonable" very much depends on your situation.

As you can see, this doesn't depend on $mean(Y)$, as would e.g. the relative error. The coefficient of determination $R^2$ tells you how good your model fit is compared to other models. The relative error, on the other hand, is often used when considering the usability of your approximation. E.g., if you know that the residuals are of the size of the means, you might have to reconsider your experimental setup.

$\endgroup$
1
  • 1
    $\begingroup$ thank you, I now understand $R^2$. $\endgroup$
    – regressor
    Jun 23 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.