# What's a reasonable mean squared error (or RMSE)?

For a given regression dataset, with target $$Y \in \mathbb{R}$$, where the training set examples have $$\mathrm{std}(Y)$$ and $$\mathrm{mean}(Y)$$, what is a mean squared error that makes the result obvious or not interesting? What would be a "reasonable MSE"? (all in relation to the standard deviation and the mean of $$Y$$).

For example, it doesn't make much sense for such MSE to be larger than the standard deviation squared (variance), because choosing a constant target to be the mean would give such error?

• Well, one option would be to, as you figured out, to use a constant - such as mean - as a simple forecast, and compare it to that. This is what $R^2$ does, compares your model to a constant forecast model (plus another constant). Jun 23, 2022 at 12:31
• Related, though not identical: stats.stackexchange.com/questions/414349/…
– mkt
Jun 23, 2022 at 12:33

You partially answered yourself. The best simple (single parameter) model that you can have to minimize squared error is to always predict the mean. In such a case, the mean squared error would be equal to the variance. What follows, if your solution is worse than this, it's really bad. Other than that, MSE or RMSE are not bounded, they can take any non-negative values. The only way to judge if the value of the metric is reasonable is by comparing it to some benchmark, like the results of another model.

You would want the MSE to be as small as possible compared to the variance. A standard measure for this is the coefficient of determination, also known as "R squared": $$R^2 = 1 - \frac{MSE}{var(Y)}.$$ But what is "reasonable" very much depends on your situation.

As you can see, this doesn't depend on $$mean(Y)$$, as would e.g. the relative error. The coefficient of determination $$R^2$$ tells you how good your model fit is compared to other models. The relative error, on the other hand, is often used when considering the usability of your approximation. E.g., if you know that the residuals are of the size of the means, you might have to reconsider your experimental setup.

• thank you, I now understand $R^2$. Jun 23, 2022 at 13:41