0
$\begingroup$

I wanted to ask whether you think that it can be useful to compare the standard deviation of the predictions (not the standard error!) in addition to other metrics like RMSE to get an idea on the spread of the data?

I wonder if it might help to indicate pontential overfitting. For instance when I have two models with roughly the same RMSE but two different SDs, shouldn't the one with the lower SD generalize better?

I am thankful for any thoughts on this!

$\endgroup$

2 Answers 2

1
$\begingroup$

I don't think this will be useful. Take for ex. the following case

enter image description here

Both predictions have similar RMSE: 3.325 and 3.30625. Very different SD: 0.302765 and 0.

Would you say that predictions 2 (with smaller SD) are better predictors?

$\endgroup$
0
$\begingroup$

In my experience that depends on the business problem you want to solve. If you need predictions that have a small standard deviation, you should really consider it when evaluating your models.

However, general speaking i would propose that if two models have the same RMSE, the model with the smaller standard deviation performs better to unseen data, assuming that the difference between the deviations is somehow significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.