0
$\begingroup$

I want to plot the RMSE of some models to compare their performance in a dataset. I'd like also to include error bars, because I know they may have very different standard deviations.

The problem is, does it even make sense to talk about standard deviation of RMSE? I thought about taking the square root of the standard deviation of the MSE, but I don't know if that's what I need.

|cite|improve this question
$\endgroup$
  • $\begingroup$ You can talk about the variance of any statistic. It is turtles all the way down. $\endgroup$ – tchakravarty Nov 19 '17 at 5:39
1
$\begingroup$

RMSE is the square root of MSE. But the answer to your question depends on if you are talking about the MSE of a predictor versus an estimator.

MSE of Estimator

MSE of an estimator is a fixed quantity, and has no variance. So it makes no sense to talk about the SD of the MSE.

Consider a special case with model, $M_1$, $Y_i = \beta$ for $n$ $iid$ observations. Suppose $E[Y_i]=\beta$ and $Var[Y_i] = \phi$, $\forall i$. An unbiased estimator is for $\beta$ is $\hat \beta = n^{-1} \sum_i Y_i$. Now $\hat \beta $ is a function of a random sample and so is random itself. If it's random, it has variance. Since it is unbiased, $MSE[\hat \beta]=Var[\hat\beta] = \phi$. So the MSE is constant.

$Var\big[MSE[\hat\beta]\big]=Var\big[\phi\big]=0$.

MSE of a Predictor

This is a function of a random sample so it is itself random and therefore has variance. Consider predictor from the model above, $M_1$

$MSE_{pred} = n^{-1} \sum_i(Y_i - \hat Y)^2 = n^{-1} \sum_i(Y_i - \hat \beta)^2 $

$\hat \beta$ is a function of data (random). $Y_i$ is the data (also random), so the whole MSE is a statistic - so is itself random. So it has variance and we can meaningfully talk about the SD of the MSE.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Right, I can calculate the SD of the MSE, but if want it to be in the same unit as the RMSE, should I just take the square root? $\endgroup$ – erickrf Nov 19 '17 at 19:05
  • 1
    $\begingroup$ In that case I'd take the square root first, then calculate the SD. That's because in general $Var(f(\hat \beta)) \neq f(Var(\hat \beta))$. In your case $f$ is the square root function. Does that address your question? $\endgroup$ – AOGSTA Nov 19 '17 at 19:51
  • $\begingroup$ If I understood correctly, you're telling me to take the SD over the absolute error... is it right? $\endgroup$ – erickrf Nov 19 '17 at 19:59
  • $\begingroup$ Not sure what you mean by absolute error. I think you want $SD(RMSE)$, but you only have $SD(MSE)$. You posit taking $\sqrt{SD(MSE)}$, but that is not what you need since $\sqrt{SD(MSE)} \neq SD(\sqrt{MSE})$ You need to calculate RMSE and then take the SD of this RMSE. You can't use the square root of the MSE's SD. $\endgroup$ – AOGSTA Nov 19 '17 at 20:11
  • $\begingroup$ Right, I understand that. As absolute error, I meant that $\sqrt{MSE} = |Y_i - \hat{Y}|$ $\endgroup$ – erickrf Nov 19 '17 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.